control_plane/attachment_service: improve Scheduler

[jcsp]

Problem

One of the major shortcuts in the initial version of this code was to construct a fresh Scheduler each time we need it, which is an O(N^2) cost as the tenant count increases.

Summary of changes

• Keep Scheduler alive through the lifetime of ServiceState
• Use IntentState as a reference tracking helper, updating Scheduler refcounts as nodes are added/removed from the intent.

There is an automated test that checks things don't get pathologically slow with thousands of shards, but it's not included in this PR because tests that implicitly test the runner node performance take some thought to stabilize/land in CI.

Checklist before requesting a review

• I have performed a self-review of my code.
• If it is a core feature, I have added thorough tests.
• Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
• If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

• Do not forget to reformat commit message to not include the above checklist

[github-actions]

2442 tests run: 2324 passed, 0 failed, 118 skipped (full report)

---

Flaky tests (2)

Postgres 15

• test_sharding_split_smoke: release

Postgres 14

• test_sharding_split_smoke: release

Code coverage (full report)

• functions: 55.8% (12941 of 23187 functions)
• lines: 82.5% (70067 of 84925 lines)

---

_{The comment gets automatically updated with the latest test results
e68a8d1 at 2024-02-19T11:35:12.366Z :recycle:}

[problame]

The transition to get_attached() could have been a preliminary PR, would have reduced the noise quite a bit.

---

What are your plans for debuggability of the reference counting?
because I'm not at all confident that

1. I spotted any possible bugs and
2. We won't have refcounting errors in the future

What alternatives to manual refcounting exist & why were they not viable?

---

I'll have to spend more time reviewing the correctness of the reference counting once I know we're definitely going to take that route versus a TBD alternative.

[problame]

Had a call about this, there will be a follow up that adds self consistency check.

Also, we currently never remove nodes from the hash map, so, worst case right now is bad scheduling decisions that can be fixed by restarting . Future bad scheduling decisions can be prevented by restarting the service, which straightens the reference counts for a moment.