pageserver: add tokio-epoll-uring slots waiters queue depth metrics

[yliang412]

In complement to neondatabase/tokio-epoll-uring#56.

Problem

We want to make tokio-epoll-uring slots waiters queue depth observable via Prometheus.

Summary of changes

• Add pageserver_tokio_epoll_uring_slots_submission_queue_depth metrics as a Histogram.
• Each thread-local tokio-epoll-uring system is given a LocalHistogram to observe the metrics.
• Keep a list of Arc<ThreadLocalMetrics> used on-demand to flush data to the shared histogram.
• Extend Collector::collect to report pageserver_tokio_epoll_uring_slots_submission_queue_depth.

Alternative Design Considered #

Since the overall idea is to let each thread observe the metrics separately and on demand aggregate the metrics, the thread_local crate may seem to be a good fit because we can use the its iterator to collect thread local metrics (Rust std library does not).

However, the implementation does not free up the thread_local storage until the ThreadLocal<T> object got destroyed. Even if thread ids are aggressively reused, we would have at least n thread local storage, where n is the maximum number of active threads throughout lifetime of the ThreadLocal<T>. This is not acceptable in our use case, as this number would be num_runtime (4) times the spawn_blocking pool size (512).

Checklist before requesting a review

• I have performed a self-review of my code.
• If it is a core feature, I have added thorough tests.
• Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
• If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

• Do not forget to reformat commit message to not include the above checklist

[github-actions]

5256 tests run: 5041 passed, 0 failed, 215 skipped (full report)

---

Flaky tests (1)

Postgres 17

• test_local_only_layers_after_crash: debug-x86-64

Code coverage* (full report)

• functions: 31.3% (7689 of 24543 functions)
• lines: 48.8% (60444 of 123907 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
143cc19 at 2024-10-25T20:12:55.440Z :recycle:}

[yliang412]

Verified metrics output locally:

Output from localhost:9898/metrics

...
pageserver_tokio_epoll_uring_slots_submission_queue_depth_bucket{le="1"} 87261
pageserver_tokio_epoll_uring_slots_submission_queue_depth_bucket{le="2"} 87261
pageserver_tokio_epoll_uring_slots_submission_queue_depth_bucket{le="4"} 87261
pageserver_tokio_epoll_uring_slots_submission_queue_depth_bucket{le="8"} 87261
pageserver_tokio_epoll_uring_slots_submission_queue_depth_bucket{le="16"} 87261
pageserver_tokio_epoll_uring_slots_submission_queue_depth_bucket{le="32"} 87261
pageserver_tokio_epoll_uring_slots_submission_queue_depth_bucket{le="64"} 87261
pageserver_tokio_epoll_uring_slots_submission_queue_depth_bucket{le="128"} 87261
pageserver_tokio_epoll_uring_slots_submission_queue_depth_bucket{le="256"} 87261
pageserver_tokio_epoll_uring_slots_submission_queue_depth_bucket{le="512"} 87261
pageserver_tokio_epoll_uring_slots_submission_queue_depth_bucket{le="1024"} 87261
pageserver_tokio_epoll_uring_slots_submission_queue_depth_bucket{le="+Inf"} 87261
pageserver_tokio_epoll_uring_slots_submission_queue_depth_sum 37
pageserver_tokio_epoll_uring_slots_submission_queue_depth_count 87261
...

Local grafana:

sum(increase(pageserver_tokio_epoll_uring_slots_submission_queue_depth_bucket[$__rate_interval])) by (le)