Support synchronize WAL in parallel #258
Labels
enhancement
New feature or request
Comments
tabokie
added a commit
that referenced
this issue
Sep 6, 2022
This is a preparing work for #258. Changes: - Move `fsync` to file handle instead of writer, so that we can call it concurrently in the future. In doing so, we have to remove the sync offset tracking and deprecate `bytes_per_sync` support - Move `prepare_write` into pipe_log. Introduce a trait `ReactiveBytes` for this purpose. - Rotate the file during write instead of doing it every write group, the timing is also delayed to the next write after exceeding limit (instead of the write that exceeds the limit). - Differentiate two types of I/O errors: unrecoverable error from fsync, and retriable error from pwrite. Only bubble error for the latter case, panic early for unrecoverable ones. - Also refactor the purge code, fix two cases where force-compact is not triggered Signed-off-by: tabokie <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Write to multiple WALs concurrently to leverage hardware parallelism.
Raft Engine provides aWriteTokenthat maps to a WAL stream underneath. IfWriteTokenis not provided, it will write to the default WAL stream.User guarantees there will be no concurrent writing to the same region using different tokens. But writes to a region are allowed to be sent to different WALs at different time.There're two ways to establish total order between logs among muliple log streams,(1) Raft Engine have awareness of Raft term of the log entries. Key-values use the term of latest log in the region.(2) Raft Engine internally manages a sequence number for each region.The text was updated successfully, but these errors were encountered: