Add possibility to set parent span context after span creation

[utr1903]

What are you trying to achieve?

Since the SpanContext is immutable, the parent context can only be set at the span creation time and not after.

This means that during the creation of the span, one already needs to have the trace context in order to create the new span with the corresponding parent context. Though, in case one does not have the context yet and to get it after the span creation, the span cannot point to the correct parent span.

Example use case

Under the normal circumstances (e.g. HTTP request or message broker), the trace context can be put into the headers by the client or the producer which the server or the consumer can then get it out and propagate the trace further. However for serverless applications talking to each other via some cloud-managed infrastructure, this is not possible.

In an example scenario as follows Lambda1 -> S3 bucket -> Lambda2, where Lambda1 puts an object into a S3 bucket that triggers Lambda2, one cannot simply propagate the context under one trace.

The trace context is to be put into the metadata of the S3 object by the Lambda1. The Lambda2 needs to have this context and start it's own spans on top it. Since S3Event do not support metadata propagation, Lambda2 has to retrieve the object from the S3 first.

This means that Lambda2 will going to have 2 initial spans:

1. it's root span that started without a parent
2. a client span to get the object from the S3 which has the root span of Lambda2 as a parent span

After 2 spans, one finally has the actual trace context. However, the 2 spans are already created and are immutable. This means that if one were to propagate the trace, these 2 spans would belong to a complete different trace that makes the trackability of the business flow a lot more difficult.

What did you expect to see?

The entire flow described above represents (or should represent) only one trace and therefore, it would be great to somehow set it up that way. This would highly ease querying of the spans belonging to a particular business logic.

[pyohannes]

1. it's root span that started without a parent
2. a client span to get the object from the S3 which has the root span of Lambda2 as a parent span

You could add a link from the root span of Lambda2 to the context from Lambda1. Adding links after span creation is now supported by the specification as an experimental feature, and thus might not have made it in all language implementations yet.

Changing the parent after span creation is problematic, as this can effectively change the trace id of the span and so would also make it necessary to also change the trace id of all children spans that might have been created before the change took place.

[jmacd]

Following @pyohannes's remark, the OpenTelemetry specification has support for adding a link to another span after it starts via AddLink(). I recognize the complexity in your use-case. You didn't mention sampling, but sampling is connected with the challenge--by the time you've got your true parent, you're two spans into a new trace. If/when OpenTelemetry tackles the problem of sampling and span links, #2918, you will have to call AddLink() twice to establish the sampling connection between the thing you want to trace and the accessory spans.

While we have heard reasons to defer various aspects of determining a span (e.g., its links, its attributes, even its name), parenting a new span means making a head-sampling decision. While we are aware of Jaeger's support for deferring a sampling decision until the first child, we believe this rarely-used feature should be accomplished in some other way.

In addition to #2918, is seems that to respect sampling and span links it is necessary to support exporting spans that were not sampled, as discussed in #2986. I will close this in favor of those two issues. I would like to see us adopt a solution in which your two accessory spans are started with a "record-only-with-span-link-sampling" configuration; your spans will not sample unless they are linked with a sampled trace, which you'll do by calling AddLink() twice.

Since I see this ultimately as a sampling-related problem, you are invited to join the weekly Sampling SIG, which is Thursdays at 4PM UTC.

[utr1903]

@jmacd @pyohannes Thanks you very much for the detailed responses! Linking can definitely tackle this use-case.
Additionally, I haven't even considered sampling at all as I was creating this issue. I'll surely keep that in mind from now on.