How to use context propagation across JVMs in batch jobs

[mamathadaram]

Am trying to use open telemetry in spark jobs. We have a driver and executor nodes running in separate jvms. I want to create tracer object and parent span created at driver node and pass this to the executor node using context propagation.

Pls let me know if we can achieve it.

[jack-berg]

There's no built in context propagator for this use case. I'm not super familiar with spark, but the key question to ask is whether you can implement the TextMapPropagator interface in a way where the trace context can be passed to the executor nodes. I'd imagine the answer is yes, but don't know the specifics.

[trask]

I'm aware of people using OpenTelemetry to monitor spark jobs, but I'm not aware of anyone yet propagating the trace across nodes. Sounds super interesting, if you end up diving in and getting something to work please share your results!