walredo spawn: avoid stalling pageserver process on CLONE_VFORK

[problame]

Problem

posix_spawn uses CLONE_VFORK under the hood, which freezes the entire parent process anyway until the child does the exec.

This issue tracks the need to avoid the latency impact of CLONE_VORK on the parent process.

Quantifying The Impact

Before jumping to implementations, make sure the problem above is actually valid.

Idea to do that: write a test program that

• serves a static http GET response using tokio + hyper
• on the same tokio runtime that uses std::process::Command to posix_spawn the true program at a steady, configurable rate

Throw wrk against the GET endpoint to measure tail latencies.

Play around with the rate of posix_spawn & observe impact on tail latencies.

We'd expect tail latencies to get worse as the spawn rate increases.

Solution Proposal, Assuming The Impact Is Material

High-level:

• Instead of spawning walredo processes directly, have a spawner process running next to pageserver.
• When Pageserver needs a walredo process, it asks the spawner to create that process.

Implementation:

• PS <> Spawner process interaction:
  • PS spawns it as a child process during startup (/proc/self/exe --mode Spawner)
  • PS and Spawner are connected via unnamed unix domain socket (socketpair())
  • Spawner has no stdio, it is for all intents and purposes a function running in a separate address space.
  • Spawner terminates if there is an error interacting with socketpair() connection.
  • When PS encounters an error on the socketpair() connection, PS SIGKILL & waits the old Spawner and starts a new one.
  • There is only one Spawner in existence at any time.
• PS <> Spawner protocol:
  • PS requests Spawner to launch a process (the request specifies binary path, argv, envp)
  • Spawner creates 3 unnamed pipes for stdin, stdout, stderr of the (grand)child
  • Spawner spawns the grand child using CLONE_PIDFD
    • Ensures only the 3 pipe file descriptors for the child make it to the child
  • Spawner responds to PS with
    • the pidfd of the child process
    • its end of the stdin, stdout, stderr file descriptors
  • (The file descriptors can be sent over the socketpair using cmsg, it looks esoteric but is used by quite lot of software.
• PS interacts with the child using the pidfd + stdin/stdout/stderr pipe ends, as it does today.

Work

Preliminary

Preview Give feedback

1. fix(walredo spawn): coalescing stalls other executors std::sync::RwLock #7310
   +0

Work

Preview Give feedback

1. Quantify the impact; rest of this list assumes it's there.
2. Implement the design

[problame]

Quantifying The Impact

Started doing that using my sample project.

The tail latency impact is clearly visible with 500k files open (3ms on my machine).
With 0 files open, it's ~300us.

experiment-posix-spawn-tail-latency-impact.zip

[problame]

However, these numbers don't justify investing time into pre-spawned walredo right now.
Closing this issue.