fix(celery): account for a chains scenario

[wantsui]

We've made a few changes to handle celery context recently:

• fix(celery): close celery.apply spans even without after_task_publish, when using apply_async #10676
• fix(celery): handle upstream celery patch and propagation during an e… #11272

In particular the goal of #10676 was to handle a scenario where a long running task may run into an exception, preventing it from closing.

Unfortunately, this scenario did not account for cases where tasks are chained and may not close until later.

See: #11479

With this PR, the sample app in #11479 would attach the celery specific span back to the root span.

This is a draft PR for now because I am waiting to see if there are unexpected failures if we remove the logic to close unclosed prerun spans in some cases. (Some more testing on this is needed).

I also need to add tests for the chains scenario.

Related to AIDM-494

Checklist

• PR author has checked that all the criteria below are met
• The PR description includes an overview of the change
• The PR description articulates the motivation for the change
• The change includes tests OR the PR description describes a testing strategy
• The PR description notes risks associated with the change, if any
• Newly-added code is easy to change
• The change follows the library release note guidelines
• The change includes or references documentation updates if necessary
• Backport labels are set (if applicable)

Reviewer Checklist

• Reviewer has checked that all the criteria below are met
• Title is accurate
• All changes are related to the pull request's stated goal
• Avoids breaking API changes
• Testing strategy adequately addresses listed risks
• Newly-added code is easy to change
• Release note makes sense to a user of the library
• If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment
• Backport labels are set in a manner that is consistent with the release branch maintenance policy

[github-actions]

CODEOWNERS have been resolved as:

ddtrace/contrib/internal/celery/app.py                                  @DataDog/apm-core-python @DataDog/apm-idm-python
ddtrace/contrib/internal/celery/signals.py                              @DataDog/apm-core-python @DataDog/apm-idm-python
tests/contrib/celery/test_integration.py                                @DataDog/apm-core-python @DataDog/apm-idm-python

[pr-commenter]

Benchmarks

Benchmark execution time: 2024-11-23 01:55:50

Comparing candidate commit bc68ed3 in PR branch wantsui/celery-chain-task with baseline commit defea4e in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 258 metrics, 0 unstable metrics.

[datadog-datadog-prod-us1]

🔴 Code Quality Violation

silent exception (...read more)

Using the pass statement in an exception block ignores the exception. Exceptions should never be ignored. Instead, the user must add code to notify an exception occurred and attempt to handle it or recover from it.