use read lock for compaction

[skyzh]

Problem

Based on the immutable storage state refactor, this is the first optimization we can do.

ref #4766 #4754

Summary of changes

We do not actually need to hold the write lock when removing the file. As long as no thread will be able to read the layer file from the layer map, we can safely remove it without taking any locks. This is done by (1) having a lock that protects that, currently pseudo_lock (2) ensure all read threads are taking the read lock (3) if we want to ensure all threads use the new version of the layer map, try acquire the write lock, and if it succeeds, all read threads are blocked and will switch to the new version.

An optimal solution would be using watermark to track lowest use version when dropping the layer object, but it's still too far so we go with this simple approach for now.

Checklist before requesting a review

• I have performed a self-review of my code.
• If it is a core feature, I have added thorough tests.
• Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
• If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

• Do not forget to reformat commit message to not include the above checklist

[github-actions]

1040 tests run: 990 passed, 0 failed, 50 skipped (full report)

---

Flaky tests (1)

Postgres 15

• test_ondemand_download_failure_to_replace[local_fs]: debug

[awestover]

seems good.

[LizardWizzard]

I wasnt closely following the whole patch set, so I'm not familiar with previous changes in the stack.

The PR looks good to me, but I'd ask for second pair of eyes from somebody familiar with compaction code cc @koivunej, @bojanserafimov

[LizardWizzard]

The concept looks quite similar to rcu family primitives. Ref https://github.com/neondatabase/neon/blob/main/libs/utils/src/simple_rcu.rs

Not asking to use it, just for reference