storcon: automatically clear Pause/Stop scheduling policies to enable detaches

[jcsp]

Problem

We saw a tenant get stuck when it had been put into Pause scheduling mode to pin it to a pageserver, then it was left idle for a while and the control plane tried to detach it.

Close: #9957

Summary of changes

• When changing policy to Detached or Secondary, set the scheduling policy to Active.
• Add a test that exercises this
• When persisting tenant shards, set their generation_pageserver to null if the placement policy is not Attached (this enables consistency checks to work, and avoids leaving state in the DB that could be confusing/misleading in future)

[github-actions]

7051 tests run: 6736 passed, 0 failed, 315 skipped (full report)

---

Flaky tests (8)

Postgres 17

• test_pageserver_gc_compaction_smoke: release-arm64

Postgres 16

• test_isolation[None]: release-x86-64
• test_timeline_archive[4]: release-arm64

Postgres 15

• test_physical_replication_config_mismatch_too_many_known_xids: release-arm64
• test_pull_timeline[True]: release-arm64

Postgres 14

• test_pg_regress[4]: release-x86-64
• test_prefetch[4]: release-x86-64
• test_pull_timeline[True]: release-x86-64

Code coverage* (full report)

• functions: 31.4% (8329 of 26510 functions)
• lines: 47.8% (65507 of 137138 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
294a941 at 2024-12-06T14:45:06.167Z :recycle:}

[VladLazar]

Code itself look fine.

---

Is this the behaviour we want though? We lose the shard scheduling policy whenever cplane detaches (e.g. idle tenant detach). This is also confusing.

Would this alternative work?
When detaching or downgrading to secondary due to external location conf, the storage controller proxies it to the pageserver irrespective of the shard policy. Pageserver handles it and we preserved the policy on the storcon.

[jcsp]

Is this the behaviour we want though? We lose the shard scheduling policy whenever cplane detaches (e.g. idle tenant detach). This is also confusing.

In common cases, yes... this is perhaps a symptom of the scheduling policies being a bit over-general as a concept. What we're using them for in practice is to use Pause to pin a tenant to a pageserver, and once we are detached, it usually makes sense to forget that pin.

When detaching or downgrading to secondary due to external location conf, the storage controller proxies it to the pageserver irrespective of the shard policy. Pageserver handles it and we preserved the policy on the storcon.

This could be fragile: it's okay if the pageserver is available, or if the pageserver is marked offline (because when it becomes available it'll get true'd up), but if the pageserver is unavailable but not yet marked offline, then it would prevent us changing the configuration of a tenant. That's arguably tolerable, but it sort of breaks the model that we can change configs in the foreground and reconcile in the background.

Then again, in the pure reconciliation loop model, it's correct to simply not detach a tenant if someone sets its placement policy to detach it but it scheduling policy is paused. For me this just points to the scheduling policies being defined in a way that doesn't quite fit how we want to use them. Maybe it should be like an pin: Option<NodeId> instead and then it intuitively makes sense that it gets cleared when you detach.

I do generally agree that there's a bit of "code smell" here... but fudging the scheduling policy is probably a less bad smell than changing the config functions to do pageserver work in the foreground instead of going via reconciliation.

[VladLazar]

Is this the behaviour we want though? We lose the shard scheduling policy whenever cplane detaches (e.g. idle tenant detach). This is also confusing.

In common cases, yes... this is perhaps a symptom of the scheduling policies being a bit over-general as a concept. What we're using them for in practice is to use Pause to pin a tenant to a pageserver, and once we are detached, it usually makes sense to forget that pin.

Oke, that's fair. Could you please update the comment for the schedulling policies to mention when they get reset?

but if the pageserver is unavailable but not yet marked offline, then it would prevent us changing the configuration of a tenant

I don't see it. We update the db and in-memory state and will reconcile when the pageserver comes back online.

[jcsp]

Added comment about usage of ShardSchedulingPolicy in 294a941

but if the pageserver is unavailable but not yet marked offline, then it would prevent us changing the configuration of a tenant

I don't see it. We update the db and in-memory state and will reconcile when the pageserver comes back online.

But the reconcile wouldn't do anything if scheduling mode was still set to Pause (i.e. for that to work we'd have to also have the change in this PR), thinking specifically of the case of a brief unavailability where we don't do a whole-node reconcile.

[VladLazar]

Added comment about usage of ShardSchedulingPolicy in 294a941

but if the pageserver is unavailable but not yet marked offline, then it would prevent us changing the configuration of a tenant

I don't see it. We update the db and in-memory state and will reconcile when the pageserver comes back online.

But the reconcile wouldn't do anything if scheduling mode was still set to Pause (i.e. for that to work we'd have to also have the change in this PR), thinking specifically of the case of a brief unavailability where we don't do a whole-node reconcile.

Right, special casing it was implied it for detach and downgrade to secondary was implied in my proposal.

[VladLazar]

I'm fine with this. I was trying to explore alternatives in the PR thread, but we don't have to block this PR.