Epic: 2024Q2/Q3 compaction work

[problame]

This epic is a tracking issue for the 2024Q2 compaction efforts.

Originally, the plan was to make the tiered compaction code from Q1 production-ready and ship it:

• Epic: productionize tiered compaction #7554

We paused the tiered compaction efforts because we keep hitting new bugs and substantial TODOs in the code.

We expect to deliver

• The test infrastructure that lets us validate the efficiency of compaction changes
  • DoD: we have a set of well-defined metrics that help us quantify a compaction algorithm's outcome (wrt request performance, space efficiency, CPU usage) and we can gather these metrics in the test suite and (possibly a subset) in production.
  • Epic: quantification of compaction algorithms #7770
• Some space efficiency improvements through GC’ing parts of layers that are no longer needed.
  • We continue using the legacy compaction and improve on the the inability to GC layers. Approach sketched in feat(pageserver): initial code sketch & test case for combined gc+compaction at gc_horizon #7948.
  • DoD: the pathological case in test_gc_feedback is no longer one
  • Epic: GC parts of layers that are no longer needed (Legacy-Enhanced Compaction) #8002

Ref child epics for GH metadata

Preview Give feedback

1. quantification of compaction algorithms #7770
   1 of 2
   a/test c/storage/pageserver +2
   
2. GC parts of layers that are no longer needed (Legacy-Enhanced Compaction) #8002
   c/storage/pageserver t/Epic +2
   

[skyzh]

closing the issue b/c all children issues are marked complete; the current Q4 work is to run the new gc-compaction in prod reliably #9114