Epic: fully eliminate latency impact of walredo spawning

[problame]

This is a spin-off from Epic pageserver: spawning walredo process is slow #6565

Problem

We currently spawn walredo processes lazily, causing the first getpage request to experience a 5ms slowdown, which is the avg spawn latency.

If the compute issues getpage requests concurrently (parallel query execution, prefetch), then all these concurrent requests coalesce on the 5ms long spawn, i.e., all experience a 5ms slowdown.

We also shut down walredo after idle time. If the compute is still up, and then issues a request, it experiences the 5ms outlier(s) again.

On moderately busy PSes (mid hundreds to low thousands of GetPage/second), the outliers introduced by this skew the p99.9X latencies.

Also, there's the impact of CLONE_VFORK (posix_spawn), which freezes the entire parent until the child does exec. See #7320

ALl of the above is definitely contributing to tail latencies; the question is: how badly, and is it easier to just fix than to quantify it.

Metrics-only solution

Only fix the metrics by counting getpage requests that needed walredo spawn seprately.
This goes in a simmilar direction as #3797

Partial Solution

Spawn walredo during basebackup. This solves the outlier on cold starts, and probably makes our p99.9X getpage stats look better.

But, cold starts + first query is dominated by cold starts, so, +5ms on the first query part won't move the needle for actual UX.

Full Solution: pre-spawned pool

Hence, to completely take walredo process spawning off the critical path, we should have a pool of pre-spawned walredo processes for use by tenants.

Note that as a preliminary, #7320 should be done fist to avoid API-level conflicts.
However, the core ideas are orthogonal:

• this issue is about the pre-spawned pool to avoid spawn latency
• issue walredo spawn: avoid stalling pageserver process on CLONE_VFORK #7320 is about avoiding process-wide freezes/stalls due to posix_spawn

Work

Preliminaries

Preview Give feedback

1. refactor(walredo): split up walredo.rs #6583
   +0
2. heavier_once_cell: switch to tokio::sync::RwLock #6589
   +0
3. refactor(walredo): don't create WalRedoManager for broken tenants #6597
   +0
4. walredo spawn: avoid stalling pageserver process on CLONE_VFORK #7320
   1 of 3
   c/storage/pageserver +1

Partial solution: spawn walredo as part of basebackup

Preview Give feedback

1. draft PR it

Full solution: pre-spawned pool

Preview Give feedback

1. refactor(walredo): use heavier_once_cell to store the process handle #6595
   +0
2. WIP: feat(walredo): pre-spawned pool of walredo processes #6596
   +0

Related

• fix(walredo spawn): coalescing stalls other executors std::sync::RwLock #7310

[problame]

The latency impact is easily demonstratable, but not even a mid-term priority.
=> close this epic to avoid it becoming stale #7320 (comment)