Add counter telemetry #2741

me-diru · 2024-08-21T19:13:21Z

Fixes #2564

I think this captures the metrics of LLM and Key-Value store

me-diru · 2024-08-21T19:16:28Z

Captured the LLM model and prompt information and kv stores get and set key information

Not sure if the key is accessible when querying on Prometheus/Grafana

cc: @calebschoepp

calebschoepp · 2024-08-21T19:29:35Z

crates/key-value/src/lib.rs

@@ -92,6 +92,9 @@ impl key_value::HostStore for KeyValueDispatch {
        store: Resource<key_value::Store>,
        key: String,
    ) -> Result<Result<Option<Vec<u8>>, Error>> {
+        // Log key value host component get feature


I like the intention behind these comments, but I think they're needlessly verbose. Directly reading spin_telemetry::counter is pretty intuitive as to what it's doing.

calebschoepp · 2024-08-21T19:31:06Z

crates/key-value/src/lib.rs

@@ -92,6 +92,9 @@ impl key_value::HostStore for KeyValueDispatch {
        store: Resource<key_value::Store>,
        key: String,
    ) -> Result<Result<Option<Vec<u8>>, Error>> {
+        // Log key value host component get feature
+        spin_telemetry::counter!(spin.key_value_get = 1, key = key);


These should all probably be monotonic_counter's b/c they're monotonically increasing.

calebschoepp · 2024-08-21T19:32:53Z

crates/key-value/src/lib.rs

@@ -102,6 +105,9 @@ impl key_value::HostStore for KeyValueDispatch {
        key: String,
        value: Vec<u8>,
    ) -> Result<Result<(), Error>> {
+        // Log key value host component set feature
+        spin_telemetry::counter!(spin.key_value_set = 1, key = key);


It occurs to me that you're adding all this to the host components which we're going to be deleting very soon. This should all probably be added to the factors which mean this would have to be a PR against the factors branch.

calebschoepp · 2024-08-21T19:33:38Z

crates/llm-remote-http/src/lib.rs

+        spin_telemetry::counter!(
+            spin.llm_infer = 1,
+            model_name = model,
+            prompt_given = prompt


I don't think we want this as an attribute. The prompt could be very large and it is also likely to be high cardinality which is not a good fit for a metric attribute.

itowlson · 2024-08-22T21:49:51Z

@me-diru is this intended to land in the factors branch, or should it be rebased off main? Currently, merging this would merge a whole bunch of unrelated factors stuff too.

calebschoepp · 2024-08-22T21:54:54Z

@me-diru is this intended to land in the factors branch, or should it be rebased off main? Currently, merging this would merge a whole bunch of unrelated factors stuff too.

This should land on factors (or main once factors merges in there).

me-diru · 2024-08-22T22:12:57Z

I changed the base to factors, and it should only reflect my code changes. Thanks for checking in @itowlson !

me-diru · 2024-08-22T22:21:05Z

@calebschoepp
I am not sure how to test the telemetry metrics in factors. For llm-compute, I think I am capturing it in the right place. However, when I run the build using

4318/v1/traces OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=h
ttp://localhost:4318/v1/metrics ../../../spin/target
/debug/spin up --runtime-config-file ../runtime-conf
ig.toml

it gives me an error of

Eror: unused runtime config key(s): llm_compute

I tried to run the factor_test.rs for the factor-llm, but it gave me another error on

error[E0277]: the trait bound `url::Url: Deserialize<'_>` is not satisfied
  --> crates/factor-llm/src/spin.rs:91:17
   |
91 | #[derive(Debug, serde::Deserialize)]
   |                 ^^^^^^^^^^^^^^^^^^ the trait `Deserialize<'_>` is not implemented for `url::Url`
   |

Which I think satisfies

On the other hand, for factor-key-value, I think it is utilizing the spin-key-value crate to do the set and get functions? So I guess the monotonic counters for key-value should suffice.

Would be great to have your input

calebschoepp · 2024-08-22T22:21:45Z

@me-diru I still see 3 commits that aren't yours that you'll want to take out of this diff.

itowlson · 2024-08-22T22:25:30Z

@me-diru

the trait Deserialize<'_> is not implemented for url::Url

You might need to enable the serde feature for the url crate. https://docs.rs/url/latest/url/#feature-serde

(This is a common pattern in Rust utility crates, where it's useful for people to be able to serialise the types, but they don't want to force a heavyweight serde dependency on people who just want to sling URL or times or whatever.)

me-diru · 2024-08-22T22:34:19Z

You might need to enable the serde feature for the url crate. https://docs.rs/url/latest/url/#feature-serde

That did the trick, tests passed :D

I am just curious how the tests passed before, though 😅 In the current case of factor-llm, we don't have to deserialize the Url?

itowlson · 2024-08-22T22:39:42Z

@me-diru It will work if any crate in the build turns the feature on. This can be cause surprises when you use your crate in a slightly different build context and the other crate that happened to make things work is no longer there and boom your code stops compiling.

This is a significant pain point for features but I gather there is not much that can be done.

calebschoepp · 2024-08-23T15:51:32Z

Are you still having errors running it @me-diru? I would need to see the runtime config you're using to help more.

me-diru · 2024-08-23T17:48:46Z

Are you still having errors running it @me-diru? I would need to see the runtime config you're using to help more.

Yes, it's still happening. When I run the same command with the latest spin cli release(2.7), it works fine.

anonymized runtime-config file

[llm_compute]
type = "remote_http"
url = "<URL>"
auth_token = "<AUTH-TOKEN>"

I checked with factors branch spin binary and the same error occurs. I don't think the metrics code is causing this one.

Maybe @lann could give more insight

lann · 2024-08-26T21:20:49Z

☝️ This hopefully fixed it.

me-diru · 2024-09-05T21:42:03Z

@calebschoepp I think it's now capturing the metrics in the new factors code!

calebschoepp · 2024-09-06T18:32:30Z

Sweet, @me-diru is this ready for a final review?

me-diru · 2024-09-21T22:14:52Z

@calebschoepp Yes! Please review :D

calebschoepp

Good work. LGTM!

Could someone from @fermyon/spin-core-maintainers enable CI and merge this?

lann · 2024-09-23T16:18:12Z

crates/key-value/src/lib.rs

@@ -98,6 +98,8 @@ impl key_value::HostStore for KeyValueDispatch {
        store: Resource<key_value::Store>,
        key: String,
    ) -> Result<Result<Option<Vec<u8>>, Error>> {
+        spin_telemetry::monotonic_counter!(spin.key_value_get = 1, key = key);


I think we might be emitting them in traces already but I'm not sure its a good idea to be sending KV store keys by default.

Danielle opened a somewhat related issue for being more careful with what we're emitting in telemetry. https://github.com/orgs/fermyon/projects/62/views/1?pane=issue&itemId=79180245

If we wanted to get fancy we could only emit these potentially sensitive values when some sort of local debug flag is set. But, for simplicities sake we're probably best of just not emitting it.

Yeah if we don't have a specific need for it lets omit until we can come up with a better approach.

vdice · 2024-09-23T19:15:27Z

Could someone from @fermyon/spin-core-maintainers enable CI and merge this?

CI didn't run as the base branch is still factors. Perhaps rebase this on main? Then CI should run.

calebschoepp · 2024-09-23T20:01:50Z

Good catch, @me-diru you'll have to point this onto main.

me-diru · 2024-09-23T23:52:30Z

The plan was to merge it to factors branch capturing metrics in the new codebase. @calebschoepp I wonder if the main branch was updated to point the PR there? 👀

I see all commits from factors branch are merged to main. I will do that, thanks! :D

me-diru · 2024-09-24T00:26:02Z

I have excluded capturing key details in the metrics. However, I was still getting some weird linting issues. It would be great to get more context on them!

calebschoepp · 2024-09-24T14:59:32Z

I have excluded capturing key details in the metrics. However, I was still getting some weird linting issues. It would be great to get more context on them!

Seems like there aren't any linting errors in CI. Are you just experiencing those linting errors locally? What are they?

me-diru · 2024-09-25T21:27:03Z

Seems like there aren't any linting errors in CI. Are you just experiencing those linting errors locally? What are they?

I think so. This is what I got.

Interesting to see it not being reflected in CI though

itowlson · 2024-09-25T21:35:20Z

@me-diru CI is currently locked to Rust 1.79. Is it possible you are on a more recent version of Rust with shiny new lints?

rylev · 2024-09-26T12:22:27Z

FWIW the lint warnings there have been fixed in #2866 so a rebase will get rid of those.

me-diru · 2024-10-02T06:05:23Z

@itowlson I am using Rust 1.81! That makes sense. Thanks for chiming in!

It's good to go now. I will rebase and get it merged @rylev

me-diru · 2024-10-02T06:22:21Z

Interesting to see most of my merge conflicts were version downgrades

calebschoepp · 2024-10-02T16:28:21Z

examples/spin-timer/Cargo.lock

+[[package]]
+name = "wit-parser"
+version = "0.217.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"


Looks like you need to fix this lockfile for linting to pass.

Signed-off-by: Rohit Dandamudi <[email protected]>

me-diru · 2024-10-03T08:16:47Z

There was a timeout error in all-integration-tests under Zig Setup. Rerunning it just to make sure.

Run goto-bus-stop/setup-zig@v2
AggregateError [ETIMEDOUT]: 
    at internalConnectMultiple (node:net:1117:1[8](https://github.com/fermyon/spin/actions/runs/11154357210/job/31003494593?pr=2741#step:4:9))
    at internalConnectMultiple (node:net:1185:5)
    at Timeout.internalConnectMultipleTimeout (node:net:1711:5)
    at listOnTimeout (node:internal/timers:575:[11](https://github.com/fermyon/spin/actions/runs/11154357210/job/31003494593?pr=2741#step:4:12))
    at process.processTimers (node:internal/timers:514:7)
    ```

calebschoepp suggested changes Aug 21, 2024

View reviewed changes

me-diru force-pushed the OTel-metric branch 2 times, most recently from cd4491a to ea08fb6 Compare August 22, 2024 21:40

me-diru force-pushed the OTel-metric branch from ea08fb6 to 8aac035 Compare August 22, 2024 22:07

me-diru changed the base branch from main to factors August 22, 2024 22:07

me-diru force-pushed the OTel-metric branch from 8aac035 to b8e58e8 Compare August 22, 2024 22:27

lann mentioned this pull request Aug 26, 2024

factors: Fix toml key tracking #2765

Merged

me-diru force-pushed the OTel-metric branch from b8e58e8 to 88dd34f Compare September 5, 2024 21:40

me-diru requested a review from calebschoepp September 5, 2024 21:40

me-diru force-pushed the OTel-metric branch from 88dd34f to 185cc10 Compare September 21, 2024 22:14

me-diru changed the title ~~Add counter telemetry(WIP)~~ Add counter telemetry Sep 21, 2024

calebschoepp approved these changes Sep 23, 2024

View reviewed changes

lann reviewed Sep 23, 2024

View reviewed changes

me-diru changed the base branch from factors to main September 23, 2024 23:55

me-diru force-pushed the OTel-metric branch 2 times, most recently from 88bc510 to 8b03d12 Compare September 24, 2024 00:24

me-diru force-pushed the OTel-metric branch from 8b03d12 to 214a4b6 Compare October 2, 2024 06:21

calebschoepp reviewed Oct 2, 2024

View reviewed changes

me-diru force-pushed the OTel-metric branch from 214a4b6 to c2a4a27 Compare October 3, 2024 01:35

Add counter telemetry

fe26af2

Signed-off-by: Rohit Dandamudi <[email protected]>

me-diru force-pushed the OTel-metric branch from c2a4a27 to fe26af2 Compare October 3, 2024 08:15

Add counter telemetry #2741

Are you sure you want to change the base?

Add counter telemetry #2741

Conversation

me-diru commented Aug 21, 2024

me-diru commented Aug 21, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

itowlson commented Aug 22, 2024

calebschoepp commented Aug 22, 2024

me-diru commented Aug 22, 2024

me-diru commented Aug 22, 2024

calebschoepp commented Aug 22, 2024

itowlson commented Aug 22, 2024 • edited Loading

me-diru commented Aug 22, 2024

itowlson commented Aug 22, 2024

calebschoepp commented Aug 23, 2024

me-diru commented Aug 23, 2024

lann commented Aug 26, 2024

me-diru commented Sep 5, 2024

calebschoepp commented Sep 6, 2024

me-diru commented Sep 21, 2024

calebschoepp left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vdice commented Sep 23, 2024

calebschoepp commented Sep 23, 2024

me-diru commented Sep 23, 2024 • edited Loading

me-diru commented Sep 24, 2024

calebschoepp commented Sep 24, 2024

me-diru commented Sep 25, 2024

itowlson commented Sep 25, 2024

rylev commented Sep 26, 2024

me-diru commented Oct 2, 2024

me-diru commented Oct 2, 2024

Choose a reason for hiding this comment

me-diru commented Oct 3, 2024

itowlson commented Aug 22, 2024 •

edited

Loading

me-diru commented Sep 23, 2024 •

edited

Loading