pageserver: add L0 compute backpressure (replace flush backpressure)

[erikgrinaker]

In #8550, we began backpressuring layer file flushes by waiting for S3 uploads after each flush. However, this has a few issues:

• It prevents upload parallelism.
• It prevents flush and upload pipelining.
• It slows down ingestion even when there is no need to backpressure.
• It doesn't directly backpressure WAL ingestion, and will build up in-memory layers.

Recall the existing compute backpressure knobs:

• max_replication_write_lag: 500 MB (based on last_received_lsn).
• max_replication_flush_lag: 10 GB (based on disk_consistent_lsn).
• max_replication_apply_lag: disabled (based on remote_consistent_lsn).

The flush backpressure, as it's currently implemented, delays disk_consistent_lsn. This means that we can have a pile-up of 10 GB of in-memory layers.

This backpressure was motivated by avoiding buildup of L0 files and the associated read-amplification, but it only does so indirectly. Instead, we should backpressure directly based on L0 size and the compaction backlog. Specifically, we should add:

• max_replication_compact_lag: X (based on remote_compacted_lsn).

We should also find appropriate values for flush_lag and/or compact_lag -- 10 GB seems way too high. But it must be high enough that the pageserver will actually run compaction before we backpressure.

Related to #5897.
Related to #5415.
Related to #8390.

[skyzh]

maybe related: #8390

[skyzh]

and https://www.notion.so/neondatabase/Backpressure-a7f66802e63244bcba0aa3a1b966d3f4

[VladLazar]

Is the suggestion here to add a new backpressure knob on the compute? Why not use the existing disabled one, max_replication_apply_lag?

[erikgrinaker]

Why not use the existing disabled one, max_replication_apply_lag?

Because that only tracks LSNs uploaded to S3, including L0 layers. So we could keep piling up L0 layers without it affecting max_replication_apply_lag. We need to backpressure based on compaction, not uploads.

Of course, we could get rid of max_replication_apply_lag if we want to. We could also repurpose it, but the naming doesn't seem accurate.

[VladLazar]

We could also repurpose it

That's what I meant. Re-purposing is quicker turnaround since it doesn't involve compute release. Some computes are really long running anyway.

[erikgrinaker]

Yeah, worth considering. Will still require a compute release though, since the compute does the mapping of remote_consistent_lsn to max_replication_apply_lag. Unless we want to repurpose remote_consistent_lsn too -- I'll have to check what else we use it for.

[erikgrinaker]

I have a prototype of disk_compacted_lsn in #10113. This contains the last LSN that is known to be compacted to L1 on local disk on each shard.

I think we need both disk_compacted_lsn and remote_consistent_lsn -- the former tracks compaction debt (and thus approximates read amplification), while the latter tracks upload debt, and I think it's reasonable to track and handle these separately, since we want to backpressure on both.

The compaction threshold is based on number of L0 files on the local shard (default 10). I think it makes sense to use the compacted LSN rather the number of L0 files to backpressure writes, because the former tracks how much work we have to do to catch up. However, it can be difficult to tune these values together, particularly with sharding.

Consider e.g. a shard that sees few writes: it may only have 2 L0 files, and thus won't trigger compaction, but as the LSN head keeps advancing on other shards this may cause backpressure on the unloaded shard without it choosing to compact. It's possible that we get away with this by simply setting a high enough compaction backpressure setting and relying on writes getting distributed across shards. I think we have the same problem with remote_consistent_lsn. I'll give it some thought.

[erikgrinaker]

Given sharding, I think it's better to look at the actual compaction debt of each shard (L0 delta sizes and possibly in-memory and frozen layers), and backpressure based on the maximum shard debt. I think we should do the same for upload debt.

These computations probably get involved enough that we should try to move them to the Safekeeper/Pageserver side rather than do them on the compute. I wrote up an issue for a single, variable backpressure signal: #10116.

[erikgrinaker]

For now, we'll just remove the flush backpressure, and revisit improved Pageserver backpressure later. I've written up a proposal in #8390.

[erikgrinaker]

The flush backpressure was removed in #10135. Compaction backpressure is tracked by #8390.