pageserver: assert that keys belong to shard

[erikgrinaker]

We've seen cases where stray keys end up on the wrong shard. This shouldn't happen. Add debug assertions to prevent this. In release builds, we should be lenient in order to handle changing key ownership policies.

Touches #9914.

[github-actions]

7066 tests run: 6757 passed, 0 failed, 309 skipped (full report)

---

Flaky tests (7)

Postgres 17

• test_physical_replication_config_mismatch_max_locks_per_transaction: release-arm64

Postgres 16

• test_pageserver_gc_compaction_smoke: release-arm64

Postgres 15

• test_prefetch[None]: release-x86-64, release-arm64

Postgres 14

• test_prefetch[None]: release-arm64
• test_pull_timeline[True]: release-arm64, release-x86-64

Code coverage* (full report)

• functions: 31.4% (8323 of 26501 functions)
• lines: 47.7% (65438 of 137048 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
c340914 at 2024-12-06T10:34:36.698Z :recycle:}

[erikgrinaker]

This keeps timing out on the v17 debug without-lfc regression tests. I wouldn't expect the added assertions here to have a significant performance impact, but I'll look into it.

[erikgrinaker]

Idk, this barely shows up on CPU profiles -- validate_lsn_order is at least two orders of magnitude more expensive.

On a successful run over on #10000, this job completed in 40 minutes. Here, it times out after an hour.

Maybe the test fails in some other way, e.g. a node crash that manifests as a timeout. Having a look at the test artifacts.

[erikgrinaker]

I'm not seeing any other issues except spurious timeouts, and I don't see how this change would be expensive enough to increase runtime from 40 to 60 minutes. Rerunning.

[erikgrinaker]

I don't know why this wasn't included in any of the CI artifacts, but I found the issue with test_isolation locally:

ERROR wal_connection_manager{tenant_id=c1ae72753459caf54556dafe583b6c54 shard_id=0204 timeline_id=31886a8ef88a0618b1d4fb9a28900f92}:connection{node_id=1}:panic{thread=walreceiver worker location=pageserver/src/tenant/timeline.rs:5897:21}: key 000000067F0000400000000A2E0100000002 does not belong on shard 0204

The test just hangs at this point, since the compute is waiting for the pageserver.

Let's see what this is.

[erikgrinaker]

It's an FSM key:

field1: 00         kind => relation key
field2: 0000067F   spcnode
field3: 00004000   dbnode
field4: 00000A2E   relnode
field5: 01         forknum => FSM_FORKNUM
field6: 00000002   

Forking off #10027.