[WIP] pageserver: use interpreted wal proto by default #9747
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
VladLazar
force-pushed
the
vlad/safekeeper-interpret-wal
branch
from
55faef6 to
474c03a
Compare
VladLazar
force-pushed
the
vlad/safekeeper-interpret-wal-test-enable
branch
from
0024fe2 to
492c656
Compare
VladLazar
force-pushed
the
vlad/safekeeper-interpret-wal
branch
from
474c03a to
55f5e89
Compare
VladLazar
force-pushed
the
vlad/safekeeper-interpret-wal-test-enable
branch
from
492c656 to
6472a73
Compare
5490 tests run: 5245 passed, 2 failed, 243 skipped (full report)Failures on Postgres 17
Failures on Postgres 15
Test coverage report is not availableThe comment gets automatically updated with the latest test results
d0d34e1 at 2024-11-18T12:20:24.217Z :recycle: |
VladLazar
force-pushed
the
vlad/safekeeper-interpret-wal
branch
from
55f5e89 to
a08f5f8
Compare
VladLazar
force-pushed
the
vlad/safekeeper-interpret-wal-test-enable
branch
from
6472a73 to
d2666e0
Compare
VladLazar
force-pushed
the
vlad/safekeeper-interpret-wal
branch
from
a08f5f8 to
c4ae5f8
Compare
VladLazar
force-pushed
the
vlad/safekeeper-interpret-wal-test-enable
branch
from
d2666e0 to
42e223e
Compare
VladLazar
force-pushed
the
vlad/safekeeper-interpret-wal
branch
from
c4ae5f8 to
25db978
Compare
VladLazar
force-pushed
the
vlad/safekeeper-interpret-wal-test-enable
branch
from
42e223e to
0b84b29
Compare
VladLazar
force-pushed
the
vlad/safekeeper-interpret-wal
branch
from
25db978 to
c1e1b25
Compare
VladLazar
force-pushed
the
vlad/safekeeper-interpret-wal-test-enable
branch
from
0b84b29 to
eb58c89
Compare
VladLazar
force-pushed
the
vlad/safekeeper-interpret-wal
branch
2 times, most recently
from
dcf30e3 to
c8b8b05
Compare
VladLazar
force-pushed
the
vlad/safekeeper-interpret-wal-test-enable
branch
from
eb58c89 to
9cdd01e
Compare
VladLazar
force-pushed
the
vlad/safekeeper-interpret-wal
branch
from
c8b8b05 to
7a7031f
Compare
VladLazar
force-pushed
the
vlad/safekeeper-interpret-wal-test-enable
branch
from
9cdd01e to
9247fd0
Compare
VladLazar
force-pushed
the
vlad/safekeeper-interpret-wal
branch
from
7a7031f to
f51ff0a
Compare
VladLazar
force-pushed
the
vlad/safekeeper-interpret-wal-test-enable
branch
from
9247fd0 to
7cfc9b8
Compare
VladLazar
force-pushed
the
vlad/safekeeper-interpret-wal
branch
from
f51ff0a to
8c4deee
Compare
VladLazar
force-pushed
the
vlad/safekeeper-interpret-wal-test-enable
branch
from
7cfc9b8 to
d0d34e1
Compare
github-merge-queue bot
pushed a commit
that referenced
this pull request
Nov 25, 2024
…#9746) ## Problem For any given tenant shard, pageservers receive all of the tenant's WAL from the safekeeper. This soft-blocks us from using larger shard counts due to bandwidth concerns and CPU overhead of filtering out the records. ## Summary of changes This PR lifts the decoding and interpretation of WAL from the pageserver into the safekeeper. A customised PG replication protocol is used where instead of sending raw WAL, the safekeeper sends filtered, interpreted records. The receiver drives the protocol selection, so, on the pageserver side, usage of the new protocol is gated by a new pageserver config: `wal_receiver_protocol`. More granularly the changes are: 1. Optionally inject the protocol and shard identity into the arguments used for starting replication 2. On the safekeeper side, implement a new wal sending primitive which decodes and interprets records before sending them over 3. On the pageserver side, implement the ingestion of this new replication message type. It's very similar to what we already have for raw wal (minus decoding and interpreting). ## Notes * This PR currently uses my [branch of rust-postgres](https://github.com/neondatabase/rust-postgres/tree/vlad/interpreted-wal-record-replication-support) which includes the deserialization logic for the new replication message type. PR for that is open [here](neondatabase/rust-postgres#32). * This PR contains changes for both pageservers and safekeepers. It's safe to merge because the new protocol is disabled by default on the pageserver side. We can gradually start enabling it in subsequent releases. * CI tests are running on #9747 ## Links Related: #9336 Epic: #9329
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
Summary of changes