perf: high read amplification

[skyzh]

Steps to reproduce

Expected result

Actual result

Given that we never merge existing delta layers, if a page is updated multiple times throughout a long time, the read cost of reconstructing the page would be high. Currently it would be reading 6 files on average and 20 files p99 to reconstruct a page.

This would be resolved as a long-term task as in #4359

Environment

Logs, links

[akashmahalik]

Sorry if I am asking dumb questions !

Is the page getting updated in-place
OR

1. A new page is created by reading the existing one. (Isn't the existing page indexed ?)
2. Update stuff in the page.
3. Point it to the new page.

Can you please clarify if the read amplification is the one I mentioned in Point No. 1 ?

What do you mean by reading files in this case ?

[skyzh]

It is like the latter one but not exactly. Reads involve finding all deltas and the base image, and reconstruct the latest page content.

[akashmahalik]

I read through the compaction epic and seems like the compaction isn’t done yet and we are trying to reconstruct the page by reading all the small files.

I have few questions

1. Do you have an index (sparse index -> key : byte offset) as metadata with a fixed size beginning of the file ? More like pointing to a block within a file. Future optimisation : that block can be compressed to reduce I/O bandwidth.
2. Might be possible the read amplification are just false positives where you go through all the files just to find nothing. Maybe a bloom filter might help in this case.

[skyzh]

Yes. Reading bloom filter is still costly in I/O.