Instrument task runs

[jeanluciano]

Adds OTEL instrumentation into the task engine so that if a user were to configure application-level telemetry, the task run engine will now emit spans for runs that capture metadata about the task run in span attributes and state changes as events on the span.

What impact will this have if I do not have telemetry configured?
https://opentelemetry.io/docs/concepts/instrumentation/libraries/#performance

Performance
OpenTelemetry API is no-op and very performant when there is no SDK in the application

This PR also sets up some testing infrastructure for testing otel instrumentation implemented by @collincchoy in #16010

Checklist

• This pull request references any related issue by including "closes <link to issue>"
  • If no issue exists and your change is not a small fix, please create an issue first.
• If this pull request adds new functionality, it includes unit tests that cover the changes
• If this pull request removes docs files, it includes redirect settings in mint.json.
• If this pull request adds functions or classes, it includes helpful docstrings.

[codspeed-hq]

CodSpeed Performance Report

Merging #15955 will not alter performance

_{Comparing jean/cloud-565-task-run-instrumentation (6f1cbfb) with main (ede23b1)}

Summary

✅ 3 untouched benchmarks

[cicdw]

a few changes that stood out to me; still need to review more closely

[zzstoatzz]

this lgtm! cc @desertaxle for a second look

[desertaxle]

Looks good overall, but I think we can clean up the code a little by moving most of the new otel functionality to the base class and using it in the separate sync and async engines. Also, we'll want some coverage on the SyncTaskRunEngine.

[zzstoatzz]

looking pretty good! a couple comments

[zzstoatzz]

Suggested change

	def get_labels_from_context(context: Optional[FlowRunContext]) -> Dict[str, Any]:
	if context is None:
	return {}
	return context.flow_run.labels

[zzstoatzz]

this is now unused

[zzstoatzz]

these mutable defaults should be optionals that default to None