pageserver: misrouted FSM key

[erikgrinaker]

Seen in #9943 (comment). An assertion shows that an FSM key was misrouted to shard 2 during test_isolation:

ERROR wal_connection_manager{tenant_id=c1ae72753459caf54556dafe583b6c54 shard_id=0204 timeline_id=31886a8ef88a0618b1d4fb9a28900f92}:connection{node_id=1}:panic{thread=walreceiver worker location=pageserver/src/tenant/timeline.rs:5897:21}: key 000000067F0000400000000A2E0100000002 does not belong on shard 0204

Key: 000000067F0000400000000A2E0100000002

field1: 00         kind => relation key
field2: 0000067F   spcnode
field3: 00004000   dbnode
field4: 00000A2E   relnode
field5: 01         forknum => FSM_FORKNUM
field6: 00000002   

[erikgrinaker]

It's SMGR_TRUNCATE_FSM here:

neon/pageserver/src/walingest.rs

Lines 587 to 601 in dcb6295

	if blkno % pg_constants::SLOTS_PER_FSM_PAGE != 0 {
	// Tail of last remaining FSM page has to be zeroed.
	// We are not precise here and instead of digging in FSM bitmap format just clear the whole page.
	modification.put_rel_page_image_zero(rel, fsm_physical_page_no)?;
	fsm_physical_page_no += 1;
	}
	// TODO: re-examine the None case here wrt. sharding; should we error?
	let nblocks = get_relsize(modification, rel, ctx).await?.unwrap_or(0);
	if nblocks > fsm_physical_page_no {
	// check if something to do: FSM is larger than truncate position
	self.put_rel_truncation(modification, rel, fsm_physical_page_no, ctx)
	.await?;
	}
	}
	if flags & pg_constants::SMGR_TRUNCATE_VM != 0 {

I think we need the same shard filtering logic as for the corresponding visibility map code here:

neon/pageserver/src/walingest.rs

Lines 621 to 633 in dcb6295

	if (trunc_byte != 0 || trunc_offs != 0)
	&& self.shard.is_key_local(&rel_block_to_key(rel, vm_page_no))
	{
	modification.put_rel_wal_record(
	rel,
	vm_page_no,
	NeonWalRecord::TruncateVisibilityMap {
	trunc_byte,
	trunc_offs,
	},
	)?;
	vm_page_no += 1;
	}