Investigate 10x perf difference among pageservers

[bojanserafimov]

Some are 10x faster than others during some moments, and perf over time fluctuates. Could be due to overload, or there could be something to improve in the pageserver. Not clear yet.

There's no strict dependency, but would be nice to get #4906 done first

[vadim2404]

cc @shanyp. It impacts computes start-up time.

[bojanserafimov]

It impacts computes start-up time.

This is more about overall get_page perf. If one pageserver is 10x faster than another, then we have 10x optimization opportunity.

Startup time also depends on pageserver (uncorrelated with overall pageserver perf), but it's unclear if that will still be the case after #4912

[shanyp]

@problame do we already have the get_page slo ?

[bojanserafimov]

I made a new dashboard using the new metrics that @problame added. Average get page latency ranges from 15us to 500us, depending on pageserver. It's possible that it's caused by difference in workloads, but the gap is stable over time, so probably not. Also there's no obvious correlation with number of requests.

[problame]

Closing this because

• staleness
• we have established the per-instance SLOs
• improvements from feat(walredo): use posix_spawn by moving close_fds() work to walredo C code #6574
  • although it likely wasn't an issue when Bojan created this issue