-
Notifications
You must be signed in to change notification settings - Fork 463
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(pageserver): newtype for dio-aligned buffer allocation #8730
Conversation
Signed-off-by: Yuchen Liang <[email protected]>
Signed-off-by: Yuchen Liang <[email protected]>
No tests were run or test report is not availableTest coverage report is not availableThe comment gets automatically updated with the latest test results
eb16aa9 at 2024-09-29T21:12:55.472Z :recycle: |
Signed-off-by: Yuchen Liang <[email protected]>
Signed-off-by: Yuchen Liang <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would feel better without having the Vec
underneath. Seems like the only convenience we get from it is the capacity()
/ len()
/ set_len()
method. I think we can implement those ourselves.
I think in an earlier sync meeting, I asked about whether you have some WIP stacked atop this PR that demonstrates how it will be used?
Signed-off-by: Yuchen Liang <[email protected]>
Signed-off-by: Yuchen Liang <[email protected]>
Co-authored-by: Christian Schwarz <[email protected]>
Signed-off-by: Yuchen Liang <[email protected]>
Part of #8130. ## Problem Currently, decompression is performed within the `read_blobs` implementation and the decompressed blob will be appended to the end of the `BytesMut` buffer. We will lose this flexibility of extending the buffer when we switch to using our own dio-aligned buffer (WIP in #8730). To facilitate the adoption of aligned buffer, we need to refactor the code to perform decompression outside `read_blobs`. ## Summary of changes - `VectoredBlobReader::read_blobs` will return `VectoredBlob` without performing decompression and appending decompressed blob. It becomes the caller's responsibility to decompress the buffer. - Added a new `BufView` type that functions as `Cow<Bytes, &[u8]>`. - Perform decompression within `VectoredBlob::read` so that people don't have to explicitly thinking about compression when using the reader interface. Signed-off-by: Yuchen Liang <[email protected]>
6b8a292
to
87fab6b
Compare
Signed-off-by: Yuchen Liang <[email protected]>
87fab6b
to
f6d0ed6
Compare
Problem
We need to use aligned buffer to enable Direct IO. Currently there is no primitive for constructing such buffer.
Summary of changes
IoBufferMut
, new-type aroundManuallyDrop<Vec<u8>>
with custom allocation and deallocation.bytes::BufMut
trait forIoBufferMut
tokio_epoll_uring::IoBuf
,tokio_epoll_uring::IoBufMut
trait forIoBufferMut
IoBufAlignedMut
to indicate alignment.Run
miri
against the unit tests, passed.The only error emitted was
unsupported operation: can't call foreign function _rjem_malloc on OS linux
, this is due to miri's own limitations to support jemalloc without theunprefixed_malloc_on_supported_platforms
feature on. Otherwise,Checklist before requesting a review
Checklist before merging