feat(pageserver): gc-compaction split job and partial scheduler #9897
+209
−20
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
arpad-m
approved these changes
Nov 26, 2024
This was referenced Nov 26, 2024
7040 tests run: 6731 passed, 0 failed, 309 skipped (full report)Flaky tests (7)Postgres 17
Postgres 16
Code coverage* (full report)
* collected from Rust tests only The comment gets automatically updated with the latest test results
4c86d0c at 2024-12-06T18:35:48.653Z :recycle: |
problame
approved these changes
Dec 5, 2024
There was a problem hiding this comment.
Wow, compaction_iteration is getting messier and messier.
As I already state in the base PR, I'm ok with the mess for now but this should be cleaned up before the end of this project.
Signed-off-by: Alex Chi Z <[email protected]>
Signed-off-by: Alex Chi Z <[email protected]>
skyzh
force-pushed
the
skyzh/schedule-partial-compaction
branch
from
2077ba4 to
15f26fe
Compare
Signed-off-by: Alex Chi Z <[email protected]>
Signed-off-by: Alex Chi Z <[email protected]>
Signed-off-by: Alex Chi Z <[email protected]>
github-merge-queue bot
pushed a commit
that referenced
this pull request
Dec 12, 2024
## Problem part of #9114, stacked PR over #9897, partially refactored to help with #10031 ## Summary of changes * gc-compaction takes `above_lsn` parameter. We only compact the layers above this LSN, and all data below the LSN are treated as if they are on the ancestor branch. * refactored gc-compaction to take `GcCompactJob` that describes the rectangular range to be compacted. * Added unit test for this case. --------- Signed-off-by: Alex Chi Z <[email protected]> Co-authored-by: Christian Schwarz <[email protected]>
github-merge-queue bot
pushed a commit
that referenced
this pull request
Dec 19, 2024
…#10044) ## Problem In #9897 we temporarily disabled the layer valid check because the current one only considers the end result of all compaction algorithms, but partial gc-compaction would temporarily produce an "invalid" layer map. part of #9114 ## Summary of changes Allow LSN splits to overlap in the slow path check. Currently, the valid check is only used in storage scrubber (background job) and during gc-compaction (without taking layer lock). Therefore, it's fine for such checks to be a little bit inefficient but more accurate. --------- Signed-off-by: Alex Chi Z <[email protected]> Co-authored-by: Arpad Müller <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
part of #9114, stacked PR over #9809
The compaction scheduler now schedules partial compaction jobs.
Summary of changes