consumption_metrics: sending previously sent values on restart #5206
Labels
c/storage/pageserver
Component: storage: pageserver
Comments
|
Solving this might lead to another issue which is uploading while executor threads are blocked. |
This was referenced Sep 13, 2023
koivunej
added a commit
that referenced
this issue
Sep 16, 2023
Write collected metrics to disk to recover previously sent metrics on
restart.
Recover the previously collected metrics during startup, send them over
at right time
- send cached synthetic size before actual is calculated
- when `last_record_lsn` rolls back on startup
- stay at last sent `written_size` metric
- send `written_size_delta_bytes` metric as 0
Add test support: stateful verification of events in python tests.
Fixes: #5206
Cc: #5175 (loggings, will be enhanced in follow-up)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Related: #3542, #5175
Maintaining an up-to-date cache, and reading the cache in the process start would allow us to send values at tight intervals before synthetic sizes have been calculated. We cannot maintain the cache by writing it once during shutdown, because in pratice pageservers often get SIGKILLed by systemd.
Alternatively the cache could be maintained by the service where pageserver sends the metrics to.
The text was updated successfully, but these errors were encountered: