Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Epic: Safekeeper S3 (WAL segment) compression #6409

Open
MMeent opened this issue Jan 19, 2024 · 0 comments
Open

Epic: Safekeeper S3 (WAL segment) compression #6409

MMeent opened this issue Jan 19, 2024 · 0 comments
Labels
c/storage/safekeeper Component: storage: safekeeper t/Epic Issue type: Epic

Comments

@MMeent
Copy link
Contributor

MMeent commented Jan 19, 2024

Motivation

WAL segments are a nice 16MB in size; and usually quite compressible. We should be able to reduce blob storage usage a lot if we compressed the segments before we upload them to S3 (or any other blobstore).

DoD

WAL segments in S3 are pre-compressed.

Implementation ideas

Add a zip/lz/zstd compressor step before uploading, and decompressor step during/after downloading.

Tasks
Preview Give feedback
No tasks being tracked yet.

Other related tasks and Epics
@MMeent MMeent added c/storage/safekeeper Component: storage: safekeeper t/Epic Issue type: Epic c/storage/wal labels Jan 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
c/storage/safekeeper Component: storage: safekeeper t/Epic Issue type: Epic
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jcsp @MMeent