bypass PageCache for InMemoryLayer + avoid Value::deser on L0 flush

[problame]

Part of Epic: Bypass PageCache for user data blocks.

Problem

InMemoryLayer still uses the PageCache for all data stored in the VirtualFile that underlies the EphemeralFile.

Background

Before this PR, EphemeralFile is a fancy and (code-bloated) buffered writer around a VirtualFile that supports blob_io.

The InMemoryLayerInner::index stores offsets into the EphemeralFile.
At those offset, we find a varint length followed by the serialized Value.

Vectored reads (get_values_reconstruct_data) are not in fact vectored - each Value that needs to be read is read sequentially.

The will_init bit of information which we use to early-exit the get_values_reconstruct_data for a given key is stored in the serialized Value, meaning we have to read & deserialize the Value from the EphemeralFile.

The L0 flushing also needs to re-determine the will_init bit of information, by deserializing each value during L0 flush.

Changes

1. Store the value length and will_init information in the InMemoryLayer::index. The EphemeralFile thus only needs to store the values.
2. For get_values_reconstruct_data:

• Use the in-memory index figures out which values need to be read. Having the will_init stored in the index enables us to do that.
• View the EphemeralFile as a byte array of "DIO chunks", each 512 bytes in size (adjustable constant). A "DIO chunk" is the minimal unit that we can read under direct IO.
• Figure out which chunks need to be read to retrieve the serialized bytes for thes values we need to read.
• Coalesce chunk reads such that each DIO chunk is only read once to serve all value reads that need data from that chunk.
• Merge adjacent chunk reads into larger EphemeralFile::read_exact_at_eof_ok of up to 128k (adjustable constant).

3. The new EphemeralFile::read_exact_at_eof_ok fills the IO buffer from the underlying VirtualFile and/or its in-memory buffer.
4. The L0 flush code is changed to use the index directly, blob_io
5. We can remove the ephemeral_file::page_caching construct now.

The get_values_reconstruct_data changes seem like a bit overkill but they are necessary so we issue the equivalent amount of read system calls compared to before this PR where it was highly likely that even if the first PageCache access was a miss, remaining reads within the same get_values_reconstruct_data call from the same EphemeralFile page were a hit.

The "DIO chunk" stuff is truly unnecessary for page cache bypass, but, since we're working on direct IO and #8719 specifically, we need to do something like this anyways in the near future.

Alternative Design

The original plan was to use the vectored_blob_io code it relies on the invariant of Delta&Image layers that index order == values order.

Further, vectored_blob_io code's strategy for merging IOs is limited to adjacent reads. However, with direct IO, there is another level of merging that should be done, specifically, if multiple reads map to the same "DIO chunk" (=alignment-requirement-sized and -aligned region of the file), then it's "free" to read the chunk into an IO buffer and serve the two reads from that buffer.
=> #8719

Testing / Performance

Correctness of the IO merging code is ensured by unit tests.

Additionally, minimal tests are added for the EphemeralFile implementation and the bit-packed InMemoryLayerIndexValue.

Performance testing results are presented below.
All pref testing done on my M2 MacBook Pro, running a Linux VM.
It's a release build without --features testing.

We see definitive improvement in ingest performance microbenchmark and an ad-hoc microbenchmark for getpage against InMemoryLayer.

baseline: commit 7c74112b2a6e23c07bfd9cc62c240cd6bbdd3bd9 origin/main
HEAD: ef1c55c52e0c313be4d302794d29534591f9cdc5

cargo bench --bench bench_ingest -- 'ingest 128MB/100b seq, no delta'

baseline

ingest-small-values/ingest 128MB/100b seq, no delta
                        time:   [483.50 ms 498.73 ms 522.53 ms]
                        thrpt:  [244.96 MiB/s 256.65 MiB/s 264.73 MiB/s]

HEAD

ingest-small-values/ingest 128MB/100b seq, no delta
                        time:   [479.22 ms 482.92 ms 487.35 ms]
                        thrpt:  [262.64 MiB/s 265.06 MiB/s 267.10 MiB/s]

We don't have a micro-benchmark for InMemoryLayer and it's quite cumbersome to add one. So, I did manual testing in neon_local.

  ./target/release/neon_local stop
  rm -rf .neon
  ./target/release/neon_local init
  ./target/release/neon_local start
  ./target/release/neon_local tenant create --set-default
  ./target/release/neon_local endpoint create foo
  ./target/release/neon_local endpoint start foo
  psql 'postgresql://[email protected]:55432/postgres'
psql (13.16 (Debian 13.16-0+deb11u1), server 15.7)

CREATE TABLE wal_test (
    id SERIAL PRIMARY KEY,
    data TEXT
);

DO $$
DECLARE
    i INTEGER := 1;
BEGIN
    WHILE i <= 500000 LOOP
        INSERT INTO wal_test (data) VALUES ('data');
        i := i + 1;
    END LOOP;
END $$;

-- => result is one L0 from initdb and one 137M-sized ephemeral-2

DO $$
DECLARE
    i INTEGER := 1;
    random_id INTEGER;
    random_record wal_test%ROWTYPE;
    start_time TIMESTAMP := clock_timestamp();
    selects_completed INTEGER := 0;
    min_id INTEGER := 1;  -- Minimum ID value
    max_id INTEGER := 100000;  -- Maximum ID value, based on your insert range
    iters INTEGER := 100000000;  -- Number of iterations to run
BEGIN
    WHILE i <= iters LOOP
        -- Generate a random ID within the known range
        random_id := min_id + floor(random() * (max_id - min_id + 1))::int;

        -- Select the row with the generated random ID
        SELECT * INTO random_record
        FROM wal_test
        WHERE id = random_id;

        -- Increment the select counter
        selects_completed := selects_completed + 1;

        -- Check if a second has passed
        IF EXTRACT(EPOCH FROM clock_timestamp() - start_time) >= 1 THEN
            -- Print the number of selects completed in the last second
            RAISE NOTICE 'Selects completed in last second: %', selects_completed;

            -- Reset counters for the next second
            selects_completed := 0;
            start_time := clock_timestamp();
        END IF;

        -- Increment the loop counter
        i := i + 1;
    END LOOP;
END $$;

./target/release/neon_local stop

baseline: commit 7c74112b2a6e23c07bfd9cc62c240cd6bbdd3bd9 origin/main

NOTICE:  Selects completed in last second: 1864
NOTICE:  Selects completed in last second: 1850
NOTICE:  Selects completed in last second: 1851
NOTICE:  Selects completed in last second: 1918
NOTICE:  Selects completed in last second: 1911
NOTICE:  Selects completed in last second: 1879
NOTICE:  Selects completed in last second: 1858
NOTICE:  Selects completed in last second: 1827
NOTICE:  Selects completed in last second: 1933

ours

NOTICE:  Selects completed in last second: 1915
NOTICE:  Selects completed in last second: 1928
NOTICE:  Selects completed in last second: 1913
NOTICE:  Selects completed in last second: 1932
NOTICE:  Selects completed in last second: 1846
NOTICE:  Selects completed in last second: 1955
NOTICE:  Selects completed in last second: 1991
NOTICE:  Selects completed in last second: 1973

NB: the ephemeral file sizes differ by ca 1MiB, ours being 1MiB smaller.

Rollout

This PR changes the code in-place and is not gated by a feature flag.

[github-actions]

3807 tests run: 3700 passed, 1 failed, 106 skipped (full report)

---

Failures on Postgres 14

• test_physical_replication: release-arm64

# Run all failed tests locally:
scripts/pytest -vv -n $(nproc) -k "test_physical_replication[release-pg14]"

Flaky tests (10)

Postgres 16

• test_change_pageserver: debug-x86-64
• test_lfc_resize: debug-x86-64
• test_location_conf_churn[2]: release-x86-64

Postgres 15

• test_upgrade_generationless_local_file_paths: release-arm64
• test_replica_start_scan_clog_crashed_xids: release-arm64
• test_pull_timeline_partial_segment_integrity: release-arm64

Postgres 14

• test_location_conf_churn[1]: release-arm64
• test_location_conf_churn[2]: release-arm64
• test_location_conf_churn[3]: release-arm64
• test_sql_regress[4]: release-arm64

Code coverage* (full report)

• functions: 32.6% (7401 of 22730 functions)
• lines: 50.8% (60037 of 118295 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
b184f77 at 2024-08-28T18:40:06.504Z :recycle:}

[problame]

@jcsp @VladLazar I think this is ready for a first review.

Wanted to get @jcsp 's eyes on it because of sharded ingest.

And @VladLazar 's eyes because of vectored get / direct IO.

@yliang412, you might also want to take another look at get_values_reconstruct_data, it evolved a bit since you last checked. But, no need to do a full review.

[VladLazar]

I focused on the write path and the new implementation of get_values_reconstruct_data. I need another pass for the rest.

[problame]

Reran benchmarks, still a slight improvement (see "edits" on PR description for prior benchmark results)

[VladLazar]

Went through all of it again minus vectored dio. Looks good; nice job!

[jcsp]

ok in principal, couple of suggestions on naming, and would like to confirm whether the TODO about strictly enforcing checkpoint/layer sizes is a risk to deploying this