Fix defusing race between Tenant::shutdown and offload_timeline

[arpad-m]

There is a race condition between Tenant::shutdown's defuse_for_drop loop and offload_timeline, where timeline offloading can insert into a tenant that is in the process of shutting down, in fact so far progressed that the defuse_for_drop has already been called.

This prevents warn log lines of the form:

offloaded timeline <hash> was dropped without having cleaned it up at the ancestor

The solution piggybacks on the offloaded_timelines lock: both the defuse loop and the offloaded timeline insertion need to acquire the lock, and we know that the defuse loop only runs after the tenant has set its TenantState to Stopping.

So if we hold the offloaded_timelines lock, and know that the TenantState is not Stopping, then we know that the defuse loop has not ran yet, and holding the lock ensures that it doesn't start running while we are inserting the offloaded timeline.

Fixes #10070

[github-actions]

7095 tests run: 6797 passed, 0 failed, 298 skipped (full report)

---

Flaky tests (7)

Postgres 17

• test_lr_with_slow_safekeeper: release-arm64
• test_timeline_archival_chaos: release-arm64

Postgres 16

• test_pgdata_import_smoke[8-1024-RelBlockSize.MULTIPLE_RELATION_SEGMENTS]: release-arm64
• test_pgdata_import_smoke[None-1024-RelBlockSize.MULTIPLE_RELATION_SEGMENTS]: release-arm64

Postgres 15

• test_pgdata_import_smoke[None-1024-RelBlockSize.MULTIPLE_RELATION_SEGMENTS]: release-arm64
• test_pg_regress[4]: release-arm64

Postgres 14

• test_pgdata_import_smoke[None-1024-RelBlockSize.MULTIPLE_RELATION_SEGMENTS]: release-arm64

Code coverage* (full report)

• functions: 31.3% (8396 of 26831 functions)
• lines: 48.0% (66653 of 138900 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
ddf3128 at 2024-12-18T18:10:07.685Z :recycle:}