storcon: refine logic for choosing AZ on tenant creation

[jcsp]

Problem

When we update our scheduler/optimization code to respect AZs properly (#9916), the choice of AZ becomes a much higher-stakes decision. We will pretty much always run a tenant in its preferred AZ, and that AZ is fixed for the lifetime of the tenant (unless a human intervenes)

Eventually, when we do auto-balancing based on utilization, I anticipate that part of that will be to automatically change the AZ of tenants if our original scheduling decisions have caused imbalance, but as an interim measure, we can at least avoid making this scheduling decision based purely on which AZ contains the emptiest node.

This is a precursor to #9947

Summary of changes

• When creating a tenant, instead of scheduling a shard and then reading its preferred AZ back, make the AZ decision first.
• Instead of choosing AZ based on which node is emptiest, use the median utilization of nodes in each AZ to pick the AZ to use. This avoids bad AZ decisions during periods when some node has very low utilization (such as after replacing a dead node)

I considered also making the selection a weighted pseudo-random choice based on utilization, but wanted to avoid destabilising tests with that for now.

[github-actions]

7051 tests run: 6725 passed, 1 failed, 325 skipped (full report)

---

Failures on Postgres 14

• test_prefetch[4]: release-arm64

# Run all failed tests locally:
scripts/pytest -vv -n $(nproc) -k "test_prefetch[release-pg14-4]"

Flaky tests (4)

Postgres 16

• test_prefetch[None]: release-arm64

Postgres 15

• test_prefetch[4]: release-x86-64
• test_prefetch[None]: release-x86-64
• test_tenant_import[None-local_fs]: release-x86-64

Test coverage report is not available

_{The comment gets automatically updated with the latest test results
ecbe8e8 at 2024-12-09T11:56:34.246Z :recycle:}