pageserver: direct I/O #8130

jcsp · 2024-06-21T15:36:54Z

Project Slack: #proj-pageserver-direct-io

Currently, we do buffered reads of data pages. Direct reads would be a better fit, because:

Pageserver data pages have an extremely low temporal locality on reads, because any repeatedly accessed pages are cached inside postgres. This makes it largely a waste of memory, which we could be using for other things.
The kernel page cache gives deceptively fast read performance on lightly loaded pageservers, making performance less consistent as pageservers are packed with larger numbers of tenants.

Tasks

Give feedback

predicting/testing perf impact of direct IO (inject delays into reads that currently hit RAM?) -- test the assumption that disks are fast enough and our bottleneck is mostly software.
changing code base to use 4k-aligned-buffers everywhere, and validate this by having VirtualFile panic!() if it sees an unaligned buffer
feature-flagged switch to issue IOs with O_DIRECT
Improve userspace PageCache monitoring, where we sort index pages & expect to have a high hit rate
Reap the benefits of no longer using so much RAM for kernel page cache: make PageCache (and any other userspace caches we choose to add) larger.
Options

Backpointer to the Slack DMs between John and Christian about this: https://neondb.slack.com/archives/D05KTCVS40H/p1718977335312439

Phase 1 Impl

Give feedback

feat(pageserver): add direct io pageserver config #8622

c/storage/pageserver t/feature
vectored read: only read each stx_dio_offset_align/stx_dio_mem_align-sized section once #8719
[WIP] feat(pageserver): dio-aligned buffer allocation #8720

c/storage/pageserver
pageserver: handle decompression outside vectored read_blobs #8942

c/storage/pageserver
feat(pageserver): newtype for dio-aligned buffer allocation #8730
pageserver: remove AdjacentVectoredReadBuilder and bump minmimum io_buffer_alignment to 512 #9175
Setup a manually managed testbench in EC2
Reminder to check for fork + direct I/O https://neondb.slack.com/archives/C07BZ38E6SD/p1723657162079879
Add aligned io buffer marker trait IoBufAligned and IoBufAlignedMut
TODO: check getpage latency
pageserver: use rolling hyperloglog to estimate page cache size needed to provide near 100% hit rate #9288

c/storage/pageserver
Options

Phase 2 Impl

Give feedback

Find and fix unaligned IOs for read
pageserver: use direct IO for delta and image layer reads #9326

c/storage/pageserver
https://github.com/neondatabase/infra/pull/2150
investigate: slot leak? tokio-epoll-uring#57
pageserver: use double buffering in BufferedWriter #9387

c/storage/pageserver
pageserver: do aligned writes for delta and image layers #9868

c/storage/pageserver
Options

Follow-ups

Give feedback

pageserver: enable direct io by default in regression tests and benchmark #9713

c/storage/pageserver
explore sharing the direct IO merge code between InMemoryLayer and vectored read #8803
pageserver: investigate whether it's a good idea to replace BlockBuf::blocks with one big buffer. #10092

c/storage/pageserver m/good_first_issue
pageserver: implement buffer bypass for aligned parts of the write #10101

c/storage/pageserver
pageserver: switching all callers of BlobWriter::write_all back to write_buffered_borrowed. #10129

c/storage/pageserver
rollout delta & image reads + in-memory layer rw to prod
rollout delta & image writes to prod
Options

The text was updated successfully, but these errors were encountered:

problame · 2024-07-09T08:46:43Z

The direct IO RFC covers both read & write path. Renamed this epic to include reads and writes. (I don't think it's significantly more work).

Part of #8130, [RFC: Direct IO For Pageserver](https://github.com/neondatabase/neon/blob/problame/direct-io-rfc/docs/rfcs/034-direct-io-for-pageserver.md) ## Description Add pageserver config for evaluating/enabling direct I/O. - Disabled: current default, uses buffered io as is. - Evaluate: still uses buffered io, but could do alignment checking and perf simulation (pad latency by direct io RW to a fake file). - Enabled: uses direct io, behavior on alignment error is configurable. Signed-off-by: Yuchen Liang <[email protected]>

yliang412 · 2024-08-12T13:26:02Z

Last week:

Drafted PS Config for Direct I/O
Setup managed testbench on EC2

This week:

Finish Simulation Mode + alignment check
Use aligned buffers on read path

problame · 2024-08-13T19:32:18Z

sync meeting:

VirtualFile wrapper
primitive for aligned buffer (can we continue to use BytesMut but with an allocation that we provide?)
vectored read planner refactor: vectored read: only read each stx_dio_offset_align/stx_dio_mem_align-sized section once #8719

yliang412 · 2024-08-19T13:41:37Z

Last week:

Worked on a couple versions of feat(pageserver): newtype for dio-aligned buffer allocation #8730.
Ext4 kernel code inspection on no extra memcpy.
Started working on vectored read: only read each stx_dio_offset_align/stx_dio_mem_align-sized section once #8719.

This week:

Finish vectored read: only read each stx_dio_offset_align/stx_dio_mem_align-sized section once #8719.

yliang412 · 2024-08-26T13:29:44Z

Last week:

Opened pageserver: do vectored read on each dio-aligned section once #8763
Benchmarked chunked vectored blob read: results

refs #6989 Problem ------- After unclean shutdown, we get restarted and read the local filesystem to make decisions on those reads. Some of the data might have not yet been fsynced when the unclean shutdown completed. Durability matters even though Pageservers are conceptually just a cache of state in S3. For example: - the cloud control plane is no control loop => pageserver responses to tenant attachmentm, etc, needs to be durable. - the storage controller does not rely on this (as much?) - we don't have layer file checksumming, so, downloaded+renamed but not fsynced layer files are technically not to be trusted - #2683 Solution -------- `syncfs` the tenants directory during startup, before we start reading from it. This is a bit overkill because we do remove some temp files (InMemoryLayer!) later during startup. Further, these temp files are particularly likely to be dirty in the kernel page cache. However, we don't want to refactor that cleanup code right now, and the dirty data on pageservers is generally not that high. Last, with [direct IO](#8130) we're going to have near-zero kernel page cache anyway quite soon.

refs #6989 Problem ------- After unclean shutdown, we get restarted, start reading the local filesystem, and make decisions based on those reads. However, some of the data might have not yet been fsynced when the unclean shutdown completed. Durability matters even though Pageservers are conceptually just a cache of state in S3. For example: - the cloud control plane is no control loop => pageserver responses to tenant attachmentm, etc, needs to be durable. - the storage controller does not rely on this (as much?) - we don't have layer file checksumming, so, downloaded+renamed but not fsynced layer files are technically not to be trusted - #2683 Solution -------- `syncfs` the tenants directory during startup, before we start reading from it. This is a bit overkill because we do remove some temp files (InMemoryLayer!) later during startup. Further, these temp files are particularly likely to be dirty in the kernel page cache. However, we don't want to refactor that cleanup code right now, and the dirty data on pageservers is generally not that high. Last, with [direct IO](#8130) we're going to have near-zero kernel page cache anyway quite soon.

Part of #8130, closes #8719. ## Problem Currently, vectored blob io only coalesce blocks if they are immediately adjacent to each other. When we switch to Direct IO, we need a way to coalesce blobs that are within the dio-aligned boundary but has gap between them. ## Summary of changes - Introduces a `VectoredReadCoalesceMode` for `VectoredReadPlanner` and `StreamingVectoredReadPlanner` which has two modes: - `AdjacentOnly` (current implementation) - `Chunked(<alignment requirement>)` - New `ChunkedVectorBuilder` that considers batching `dio-align`-sized read, the start and end of the vectored read will respect `stx_dio_offset_align` / `stx_dio_mem_align` (`vectored_read.start` and `vectored_read.blobs_at.first().start_offset` will be two different value). - Since we break the assumption that blobs within single `VectoredRead` are next to each other (implicit end offset), we start to store blob end offsets in the `VectoredRead`. - Adapted existing tests to run in both `VectoredReadCoalesceMode`. - The io alignment can also be live configured at runtime. Signed-off-by: Yuchen Liang <[email protected]>

…lush (#8537) Part of [Epic: Bypass PageCache for user data blocks](#7386). # Problem `InMemoryLayer` still uses the `PageCache` for all data stored in the `VirtualFile` that underlies the `EphemeralFile`. # Background Before this PR, `EphemeralFile` is a fancy and (code-bloated) buffered writer around a `VirtualFile` that supports `blob_io`. The `InMemoryLayerInner::index` stores offsets into the `EphemeralFile`. At those offset, we find a varint length followed by the serialized `Value`. Vectored reads (`get_values_reconstruct_data`) are not in fact vectored - each `Value` that needs to be read is read sequentially. The `will_init` bit of information which we use to early-exit the `get_values_reconstruct_data` for a given key is stored in the serialized `Value`, meaning we have to read & deserialize the `Value` from the `EphemeralFile`. The L0 flushing **also** needs to re-determine the `will_init` bit of information, by deserializing each value during L0 flush. # Changes 1. Store the value length and `will_init` information in the `InMemoryLayer::index`. The `EphemeralFile` thus only needs to store the values. 2. For `get_values_reconstruct_data`: - Use the in-memory `index` figures out which values need to be read. Having the `will_init` stored in the index enables us to do that. - View the EphemeralFile as a byte array of "DIO chunks", each 512 bytes in size (adjustable constant). A "DIO chunk" is the minimal unit that we can read under direct IO. - Figure out which chunks need to be read to retrieve the serialized bytes for thes values we need to read. - Coalesce chunk reads such that each DIO chunk is only read once to serve all value reads that need data from that chunk. - Merge adjacent chunk reads into larger `EphemeralFile::read_exact_at_eof_ok` of up to 128k (adjustable constant). 3. The new `EphemeralFile::read_exact_at_eof_ok` fills the IO buffer from the underlying VirtualFile and/or its in-memory buffer. 4. The L0 flush code is changed to use the `index` directly, `blob_io` 5. We can remove the `ephemeral_file::page_caching` construct now. The `get_values_reconstruct_data` changes seem like a bit overkill but they are necessary so we issue the equivalent amount of read system calls compared to before this PR where it was highly likely that even if the first PageCache access was a miss, remaining reads within the same `get_values_reconstruct_data` call from the same `EphemeralFile` page were a hit. The "DIO chunk" stuff is truly unnecessary for page cache bypass, but, since we're working on [direct IO](#8130) and #8719 specifically, we need to do _something_ like this anyways in the near future. # Alternative Design The original plan was to use the `vectored_blob_io` code it relies on the invariant of Delta&Image layers that `index order == values order`. Further, `vectored_blob_io` code's strategy for merging IOs is limited to adjacent reads. However, with direct IO, there is another level of merging that should be done, specifically, if multiple reads map to the same "DIO chunk" (=alignment-requirement-sized and -aligned region of the file), then it's "free" to read the chunk into an IO buffer and serve the two reads from that buffer. => #8719 # Testing / Performance Correctness of the IO merging code is ensured by unit tests. Additionally, minimal tests are added for the `EphemeralFile` implementation and the bit-packed `InMemoryLayerIndexValue`. Performance testing results are presented below. All pref testing done on my M2 MacBook Pro, running a Linux VM. It's a release build without `--features testing`. We see definitive improvement in ingest performance microbenchmark and an ad-hoc microbenchmark for getpage against InMemoryLayer. ``` baseline: commit 7c74112 origin/main HEAD: ef1c55c ``` <details> ``` cargo bench --bench bench_ingest -- 'ingest 128MB/100b seq, no delta' baseline ingest-small-values/ingest 128MB/100b seq, no delta time: [483.50 ms 498.73 ms 522.53 ms] thrpt: [244.96 MiB/s 256.65 MiB/s 264.73 MiB/s] HEAD ingest-small-values/ingest 128MB/100b seq, no delta time: [479.22 ms 482.92 ms 487.35 ms] thrpt: [262.64 MiB/s 265.06 MiB/s 267.10 MiB/s] ``` </details> We don't have a micro-benchmark for InMemoryLayer and it's quite cumbersome to add one. So, I did manual testing in `neon_local`. <details> ``` ./target/release/neon_local stop rm -rf .neon ./target/release/neon_local init ./target/release/neon_local start ./target/release/neon_local tenant create --set-default ./target/release/neon_local endpoint create foo ./target/release/neon_local endpoint start foo psql 'postgresql://[email protected]:55432/postgres' psql (13.16 (Debian 13.16-0+deb11u1), server 15.7) CREATE TABLE wal_test ( id SERIAL PRIMARY KEY, data TEXT ); DO $$ DECLARE i INTEGER := 1; BEGIN WHILE i <= 500000 LOOP INSERT INTO wal_test (data) VALUES ('data'); i := i + 1; END LOOP; END $$; -- => result is one L0 from initdb and one 137M-sized ephemeral-2 DO $$ DECLARE i INTEGER := 1; random_id INTEGER; random_record wal_test%ROWTYPE; start_time TIMESTAMP := clock_timestamp(); selects_completed INTEGER := 0; min_id INTEGER := 1; -- Minimum ID value max_id INTEGER := 100000; -- Maximum ID value, based on your insert range iters INTEGER := 100000000; -- Number of iterations to run BEGIN WHILE i <= iters LOOP -- Generate a random ID within the known range random_id := min_id + floor(random() * (max_id - min_id + 1))::int; -- Select the row with the generated random ID SELECT * INTO random_record FROM wal_test WHERE id = random_id; -- Increment the select counter selects_completed := selects_completed + 1; -- Check if a second has passed IF EXTRACT(EPOCH FROM clock_timestamp() - start_time) >= 1 THEN -- Print the number of selects completed in the last second RAISE NOTICE 'Selects completed in last second: %', selects_completed; -- Reset counters for the next second selects_completed := 0; start_time := clock_timestamp(); END IF; -- Increment the loop counter i := i + 1; END LOOP; END $$; ./target/release/neon_local stop baseline: commit 7c74112 origin/main NOTICE: Selects completed in last second: 1864 NOTICE: Selects completed in last second: 1850 NOTICE: Selects completed in last second: 1851 NOTICE: Selects completed in last second: 1918 NOTICE: Selects completed in last second: 1911 NOTICE: Selects completed in last second: 1879 NOTICE: Selects completed in last second: 1858 NOTICE: Selects completed in last second: 1827 NOTICE: Selects completed in last second: 1933 ours NOTICE: Selects completed in last second: 1915 NOTICE: Selects completed in last second: 1928 NOTICE: Selects completed in last second: 1913 NOTICE: Selects completed in last second: 1932 NOTICE: Selects completed in last second: 1846 NOTICE: Selects completed in last second: 1955 NOTICE: Selects completed in last second: 1991 NOTICE: Selects completed in last second: 1973 ``` NB: the ephemeral file sizes differ by ca 1MiB, ours being 1MiB smaller. </details> # Rollout This PR changes the code in-place and is not gated by a feature flag.

yliang412 · 2024-09-02T12:34:53Z

Last week:

Merged and perf validated pageserver: do vectored read on each dio-aligned section once #8763 (on preprod), decided to go with io_buffer_alignnment=512.

This week:

Polish feat(pageserver): newtype for dio-aligned buffer allocation #8730 with a WIP stacked on top to show usage
Rethink vectored_blob_io buffer interaction (where to do decompression + where to deserialize value?)
Deploy pageserver: do vectored read on each dio-aligned section once #8763 to prod region: ap-southeast-2 and ap-southeast-1
- https://github.com/neondatabase/infra/pull/1879

problame · 2024-09-04T14:45:02Z

Sync meeting:

ship 4c0a61d as a preliminary PR, to land in prod as part of next week's release
- retain .freeze() and .slice() for uncompressed buffers
- additional allocation for compressed buffers is ok
- review from @problame and @arpad-m
continue feat(pageserver): newtype for dio-aligned buffer allocation #8730
- actually do O_DIRECT (quick and dirty)
- see if it works functionally (local test bench, regression test suite run)
- some basic pagebench getpage-at-latest-lsn benchmarking
  - 100% PS pagecache hit rate, resize like prod config, or resize further and take a note
  - 100% virtualfile fd cache hit rate, resize like prod config
- fix until not failures
- => this demonstrates we can do O_DIRECT on the read path without errors, and we have a performance ballpark

Next steps after that:

InMemoryLayer direct IO for read of tail end
- develop separately
- then cherry-pick into feat(pageserver): newtype for dio-aligned buffer allocation #8730
slice up feat(pageserver): newtype for dio-aligned buffer allocation #8730 and ship it to prod in a way that allows safe rollout

yliang412 · 2024-09-16T13:26:06Z

Last week:

deployed https://github.com/neondatabase/infra/pull/1927
This week:
Ship pageserver: handle decompression outside vectored read_blobs #8942 (a few final touches).
Continue feat(pageserver): newtype for dio-aligned buffer allocation #8730 with O_DIRECT enabled, make sure it works functionally.

Part of #8130. ## Problem Currently, decompression is performed within the `read_blobs` implementation and the decompressed blob will be appended to the end of the `BytesMut` buffer. We will lose this flexibility of extending the buffer when we switch to using our own dio-aligned buffer (WIP in #8730). To facilitate the adoption of aligned buffer, we need to refactor the code to perform decompression outside `read_blobs`. ## Summary of changes - `VectoredBlobReader::read_blobs` will return `VectoredBlob` without performing decompression and appending decompressed blob. It becomes the caller's responsibility to decompress the buffer. - Added a new `BufView` type that functions as `Cow<Bytes, &[u8]>`. - Perform decompression within `VectoredBlob::read` so that people don't have to explicitly thinking about compression when using the reader interface. Signed-off-by: Yuchen Liang <[email protected]>

yliang412 · 2024-09-25T14:51:42Z

Sync Meeting:

Continue feat(pageserver): newtype for dio-aligned buffer allocation #8730 with O_DIRECT enabled, make sure it works functionally.
Should start to increase page cache size, targeting 100% PS pagecache hit rate.

…buffer_alignment to 512 (#9175) Part of #8130 ## Problem After deploying https://github.com/neondatabase/infra/pull/1927, we shipped `io_buffer_alignment=512` to all prod region. The `AdjacentVectoredReadBuilder` code path is no longer taken and we are running pageserver unit tests 6 times in the CI. Removing it would reduce the test duration by 30-60s. ## Summary of changes - Remove `AdjacentVectoredReadBuilder` code. - Bump the minimum `io_buffer_alignment` requirement to at least 512 bytes. - Use default `io_buffer_alignment` for Rust unit tests. Signed-off-by: Yuchen Liang <[email protected]>

…buffer_alignment to 512 (#9175) Part of #8130 ## Problem After deploying neondatabase/infra#1927, we shipped `io_buffer_alignment=512` to all prod region. The `AdjacentVectoredReadBuilder` code path is no longer taken and we are running pageserver unit tests 6 times in the CI. Removing it would reduce the test duration by 30-60s. ## Summary of changes - Remove `AdjacentVectoredReadBuilder` code. - Bump the minimum `io_buffer_alignment` requirement to at least 512 bytes. - Use default `io_buffer_alignment` for Rust unit tests. Signed-off-by: Yuchen Liang <[email protected]>

yliang412 · 2024-09-30T13:23:49Z

Last week:

Merged pageserver: handle decompression outside vectored read_blobs #8942
Worked on use IoBufferMut for delta/image layers + pagecache #9169 with O_DIRECT enabled for delta + image layers. Passed all tests.
Setup new testbench.

This week:

Doubling pagecache size, to be deployed as part of this week's release: https://github.com/neondatabase/infra/pull/1961
Running pagebench on ec2-testbench with prodlike pageserver config. Compare that to buffered io.
Plan and if needed implement pagecache working set estimation
- estimate what its last-N-minutes memory would need to be for a 100% page cache hit rate.
- Slack Discussion

yliang412 · 2024-10-04T18:44:31Z

Sync Meeting:

Read Path Direct IO Deployment Plan

Ship VirtualFile changes that allows us to use direct IO for reads but still use buffered IO for writes pageserver: add direct io config to virtual file #9214.
Get rid of io_buffer_alignment config, always use 512 for alignment and chunk size.
Finish switching in-memory layer to use direct IO. Ship the direct IO read path.

Page cache

Double page cache size again in next week's release.
Implement Rolling HyperLogLog next week.
- think through compaction
Watch page cache hit rate and relationship with getpage@lsn latency

yliang412 · 2024-10-14T13:27:23Z

Last week:

Finished implementing Direct IO for image / delta layer read path, opened pageserver: use direct IO for delta and image layer reads #9326
Benchmarked using Direct IO for InMemoryLayer: https://www.notion.so/neondatabase/Benchmark-using-Direct-IO-for-InMemoryLayer-2024-10-10-11bf189e00478008aa12c912ff7b8506?pvs=4
- Performance impact on ingestion is primarily on large values if buffered writer is given a bigger buffer.
- BufferedWriter has some tech debt: flushing is done inline, blocking the next write.
https://github.com/neondatabase/infra/pull/2005
Increasing page cache size is likely not the cause of increased getpage latency
- Slack Discussion

This week:

Merge pageserver: use direct IO for delta and image layer reads #9326, separate adhoc deployment to preprod nightly prodlike cloudebench run.
Better BufferedWriter implementation with double buffering, then ship Direct IO for inmemory layer R/W and download layer (write code path that uses BufferedWriter)

yliang412 · 2024-10-21T13:26:20Z

Last week:

For pageserver: use direct IO for delta and image layer reads #9326, There was always a few pg17 debug tests that does pass for both buffered and direct IO. showing pageserver has not started yet... Investigate if it is just flake.
expose waiters queue depth metrics tokio-epoll-uring#56
- future: Possibly use thread_local crate? Benefit: have a nice iterator of thread_local values.
Prototyped a double channel based buffered writer (one task for writing, one task for flushing).

This week:

Merge pageserver: use direct IO for delta and image layer reads #9326
turn the prototyped buffered writer into a PR.

Part of #8130 ## Problem Pageserver previously goes through the kernel page cache for all the IOs. The kernel page cache makes light-loaded pageserver have deceptive fast performance. Using direct IO would offer predictable latencies of our virtual file IO operations. In particular for reads, the data pages also have an extremely low temporal locality because the most frequently accessed pages are cached on the compute side. ## Summary of changes This PR enables pageserver to use direct IO for delta layer and image layer reads. We can ship them separately because these layers are write-once, read-many, so we will not be mixing buffered IO with direct IO. - implement `IoBufferMut`, an buffer type with aligned allocation (currently set to 512). - use `IoBufferMut` at all places we are doing reads on image + delta layers. - leverage Rust type system and use `IoBufAlignedMut` marker trait to guarantee that the input buffers for the IO operations are aligned. - page cache allocation is also made aligned. _* in-memory layer reads and the write path will be shipped separately._ ## Testing Integration test suite run with O_DIRECT enabled: #9350 ## Performance We evaluated performance based on the `get-page-at-latest-lsn` benchmark. The results demonstrate a decrease in the number of IOps, no sigificant change in the latency mean, and an slight improvement on the p99.9 and p99.99 latencies. [Benchmark](https://www.notion.so/neondatabase/Benchmark-O_DIRECT-for-image-and-delta-layers-2024-10-01-112f189e00478092a195ea5a0137e706?pvs=4) ## Rollout We will add `virtual_file_io_mode=direct` region by region to enable direct IO on image + delta layers. Signed-off-by: Yuchen Liang <[email protected]>

yliang412 · 2024-11-11T14:20:39Z

Last week:

https://github.com/neondatabase/infra/pull/2286
Implemented pageserver: make BufferedWriter do double-buffering #9693, need to fix some unit tests.

This week:

pause deployment until perf regressions are understood.
pageserver: enable direct io by default in regression tests and benchmark #9713
Make pageserver: make BufferedWriter do double-buffering #9693 mergable.
- pass all unit/regression tests
- run bench_ingest to compare results.

yliang412 · 2024-11-25T06:12:31Z

Last week:

test and benchmark double-buffered writer
- performance improved for small values.
- performance regressed for large values, likely due to being CPU bound + disk write latency.
  - not as simple as turning on fsync for benchmarks. We don't fsync for in-memory layers in prod.

This week:

Merge pageserver: make BufferedWriter do double-buffering #9693, without turning on O_DIRECT.
Start working on pageserver: do aligned writes for delta and image layers #9868.

yliang412 · 2024-12-02T14:31:15Z

Last week:

Running more benchmarks using Erik's bulk insertion benchmark (test_ingest_insert_bulk) with a focus on pageserver ingestion performance (the "recovery" time): Result
- Absolute throughput are faster on hetzner box (I feel this is expected)
- Double buffering with buffered IO always improves performance.
- Double buffering with direct IO shows performance improvement on hetzner box, but regression with im4gn.2xlarge

This week:

Do more verification and at least deploy buffered writer with buffered io: pageserver: make BufferedWriter do double-buffering #9693
pageserver: do aligned writes for delta and image layers #9868:
- refactor blob writer, separate disk write logic from compression logic.

yliang412 · 2024-12-09T14:23:45Z

Last week:

Merged pageserver: make BufferedWriter do double-buffering #9693, following double-buffering / direct io rollout plan in Notion
image/delta layer writes O_DIRECT implementation work on https://github.com/neondatabase/neon/tree/yuchen/direct-io-delta-image-layer-write

This week:

Discuss direct io rollout plan again
Finish as much of pageserver: do aligned writes for delta and image layers #9868 as possible.
Document and knowledge transfer.

yliang412 · 2024-12-13T16:19:19Z

pageserver: use direct io for delta + image layer writes #10063 functionalities should be done, Needs a bench_ingest like benchmark for image writes.
- See pageserver: use direct io for delta + image layer writes #10063 (review)
Created follow-up at pageserver: direct I/O #8130 (comment)

jcsp added t/feature Issue type: feature, for new features or requests c/storage/pageserver Component: storage: pageserver labels Jun 21, 2024

problame mentioned this issue Jul 2, 2024

RFC: direct IO for Pageserver #8240

Draft

problame changed the title ~~pageserver: direct I/O for reads~~ pageserver: direct I/O Jul 9, 2024

yliang412 self-assigned this Jul 29, 2024

yliang412 mentioned this issue Aug 7, 2024

feat(pageserver): add direct io pageserver config #8622

Merged

5 tasks

problame mentioned this issue Aug 15, 2024

bypass PageCache for InMemoryLayer + avoid Value::deser on L0 flush #8537

Merged

yliang412 mentioned this issue Aug 20, 2024

pageserver: do vectored read on each dio-aligned section once #8763

Merged

5 tasks

problame mentioned this issue Aug 26, 2024

pageserver: startup: ensure local disk state is durable #8835

Merged

yliang412 mentioned this issue Aug 28, 2024

DO NOT MERGE: cherry pick https://github.com/neondatabase/neon/pull/8763 onto release branch for testing in pre-prod #8857

Closed

problame self-assigned this Sep 9, 2024

yliang412 mentioned this issue Sep 23, 2024

pageserver: handle decompression outside vectored read_blobs #8942

Merged

5 tasks

yliang412 mentioned this issue Sep 26, 2024

pageserver: remove AdjacentVectoredReadBuilder and bump minmimum io_buffer_alignment to 512 #9175

Merged

5 tasks

yliang412 mentioned this issue Oct 9, 2024

pageserver: use direct IO for delta and image layer reads #9326

Merged

5 tasks

jcsp unassigned yliang412 Dec 16, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pageserver: direct I/O #8130

pageserver: direct I/O #8130

jcsp commented Jun 21, 2024 •

edited by yliang412

Loading

Tasks

Phase 1 Impl

Phase 2 Impl

Follow-ups

problame commented Jul 9, 2024

yliang412 commented Aug 12, 2024

problame commented Aug 13, 2024 •

edited

Loading

yliang412 commented Aug 19, 2024

yliang412 commented Aug 26, 2024 •

edited

Loading

yliang412 commented Sep 2, 2024 •

edited

Loading

problame commented Sep 4, 2024

yliang412 commented Sep 16, 2024

yliang412 commented Sep 25, 2024

yliang412 commented Sep 30, 2024 •

edited

Loading

yliang412 commented Oct 4, 2024 •

edited

Loading

yliang412 commented Oct 14, 2024 •

edited

Loading

yliang412 commented Oct 21, 2024 •

edited

Loading

yliang412 commented Nov 11, 2024

yliang412 commented Nov 25, 2024 •

edited

Loading

yliang412 commented Dec 2, 2024 •

edited

Loading

yliang412 commented Dec 9, 2024 •

edited

Loading

yliang412 commented Dec 13, 2024

pageserver: direct I/O #8130

pageserver: direct I/O #8130

Comments

jcsp commented Jun 21, 2024 • edited by yliang412 Loading

Tasks

Phase 1 Impl

Phase 2 Impl

Follow-ups

problame commented Jul 9, 2024

yliang412 commented Aug 12, 2024

problame commented Aug 13, 2024 • edited Loading

yliang412 commented Aug 19, 2024

yliang412 commented Aug 26, 2024 • edited Loading

yliang412 commented Sep 2, 2024 • edited Loading

problame commented Sep 4, 2024

yliang412 commented Sep 16, 2024

yliang412 commented Sep 25, 2024

yliang412 commented Sep 30, 2024 • edited Loading

yliang412 commented Oct 4, 2024 • edited Loading

yliang412 commented Oct 14, 2024 • edited Loading

yliang412 commented Oct 21, 2024 • edited Loading

yliang412 commented Nov 11, 2024

yliang412 commented Nov 25, 2024 • edited Loading

yliang412 commented Dec 2, 2024 • edited Loading

yliang412 commented Dec 9, 2024 • edited Loading

yliang412 commented Dec 13, 2024

jcsp commented Jun 21, 2024 •

edited by yliang412

Loading

problame commented Aug 13, 2024 •

edited

Loading

yliang412 commented Aug 26, 2024 •

edited

Loading

yliang412 commented Sep 2, 2024 •

edited

Loading

yliang412 commented Sep 30, 2024 •

edited

Loading

yliang412 commented Oct 4, 2024 •

edited

Loading

yliang412 commented Oct 14, 2024 •

edited

Loading

yliang412 commented Oct 21, 2024 •

edited

Loading

yliang412 commented Nov 25, 2024 •

edited

Loading

yliang412 commented Dec 2, 2024 •

edited

Loading

yliang412 commented Dec 9, 2024 •

edited

Loading