How to use exemplars?

[stmlange]

Hello Opentelemetry Comminuty!
Thank you for this great project and I'm confident what I'm about to ask is documented somewhere, but after searching for quite some time I couldn't find any additional references.

In a nutshell I have a multithreaded application that now should be traced with OpenTelemetry. So far no issue, but now metrics come into play. The application may launch sub-processes where I want the application should add metrics to the sub-process. Each sub-process get's it's own Trace. However I'm now a bit stuck how this metric <-> trace relation ship is achieved concretely.
The documentation mentions [exemplars|https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/data-model.md#exemplars], but when using the [DoubleCounterExample|https://github.com/open-telemetry/opentelemetry-java-examples/blob/main/metrics/src/main/java/io/opentelemetry/example/metrics/DoubleCounterExample.java] with a OpenTelemetryRule for testing I can't see the connection made between the metric and the trace.

I'm close to just pass on a span_id and trace_id as additional attribute to the meter.buildWithCallback, but this really feels wrong now.
Could someone perhaps point me, or provide me concrete examples on how this metric <-> trace relation ship is achieved? I would like to also have automated tests using the OpenTelemetryRule that ensure that relationship works (e.g. my code is using OTEL correctly).

Thanks for any insights!

[trask]

hi @stmlange, I think exemplars only make sense for synchronous instruments (e.g. http.server.request.duration) since you can use the "active context" when recording those metrics

[stmlange]

Hello, thanks! When exemplars do not make sense in this case, could you perhaps elaborate how one would achieve a connection between traces <-> metrics? The only option I see currently is to encode the span and trace id as additional attribute to the metric (I guess that would be custom code? If so I'm surprised there is no OTEL util code or something, since I can't believe this would be the first use-case).

Thanks!

[trask]

hi @stmlange, this is a good point, in your case where you have a JVM process dedicated to a single trace, exemplars on async instruments could make sense

probably i'd suggest extracting the Context via W3C propagator at the JVM startup, and then passing that Context to your async callbacks

semi-related, check out open-telemetry/opentelemetry-specification#740

[stmlange]

Hello,
thanks again for your feedback!
I'm honestly a bit surprised by your commentary. I guess let me take a step back and ask my initial question somewhat differently to ensure that we understand each other:
In a nutshell I want to use OpenTelemetry to trace and track metrics for an application (and eventually logs). For me this can be rephrased as I want to understand what parts of the application take what amount of time and what amount of memory/cpu (and eventually collect more details about the failures that are encoded in the log).

For me OpenTelemetry boils down to a system that allows me to add traces/metrics/logs to "chase" down where the actual code is residing that caused the trace/metric/log. Or in other words to find the potential bottleneck for time and why is the bottleneck there (e.g. not enough memory) and further does the application properly uses the resources allocated (memory and cpu efficiency). The time is captured by traces, the memory/cpu stuff are captured by metrics. So in the end of the application I have a bunch of traces and metrics, but is there no automatic way to connect the metric with the trace (or the log with a trace)? That is the part that feels really off for me.

When I'm understanding you correctly you are suggesting to manually extract my trace_id and span_id (fine I guess) and attach them manually as extra attributes to the metrics (somewhat following the idea of https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/data-model.md#exemplars).

When following the initial example of https://github.com/open-telemetry/opentelemetry-java-examples/blob/main/metrics/src/main/java/io/opentelemetry/example/metrics/DoubleCounterExample.java this essentially means:
Instead of a:

diskSpaceCounter.add(
    (double) file.length() / 1_000_000,
    Attributes.of(
        FILE_EXTENSION_KEY, extension));

one would now do roughtly:

diskSpaceCounter.add(
    (double) file.length() / 1_000_000,
    Attributes.of(
        FILE_EXTENSION_KEY, extension,
        TRACEPARENT, traceparent,
        TRACESTATE, tracestate));

with the extraction of traceparent and tracestate using:

private static final TextMapSetter<Map<String, String>> TEXT_MAP_SETTER = Map::put;
....
final Context context = Context.current();
final Map<String, String> contextMap = new HashMap<>();
        GlobalOpenTelemetry.get()
                .getPropagators()
                .getTextMapPropagator()
                .inject(context, contextMap, TEXT_MAP_SETTER);
final String traceparent = contextMap.getIgnoreCase("TRACEPARENT");
final String tracestate = contextMap.getIgnoreCase("TRACESTATE");

Of course what is injected as extra attribute would depend on the TraceContext (in my case I'm using the W3CTraceContext).

Does it really sound correct that this would need to be done manually? If no where is the flaw in my logic / thinking? If yes, where could I file a request to have that done automatically by OpenTelemetry (since I believe this is a fundamental thing to be able to connect your different traces/metrics/logs)?

[trask]

if you use Context.makeCurrent() and use synchronous instruments then exemplars will be captured automatically