Changefeed Performance

Changefeed Performance

The overall throughput and latency of Replicator is driven by the performance of the source changefeed and the target's ability to accept writes. There are a variety of configuration knobs that can be tweaked.

Operators with more than one changefeed should consider the use of changefeed metrics labels.

Low-latency configuration

The configuration shown below trades higher source cluster CPU use and increased likelihood of transaction refreshes for lower commit-to-commit latency. Cluster tuning is workload-specific, so these settings are not necessarily correct or useful for all deployments.

-- No need to set in 24.1 and after because it's now enabled by default.
-- https://www.cockroachlabs.com/docs/releases/v24.1#v24-1-0-alpha-1-operational-changes
SET CLUSTER SETTING kv.rangefeed.enabled = true;

-- Reducing the cluster's closed timestamp interval allows the resolved timestamps to be
-- closer to the current wall time. The tradeoff with decreasing the closed timestamp
-- interval is that SQL transactions have a higher chance of requiring a read-refresh,
-- leading to increased CPU activity within the cluster.
SET CLUSTER SETTING kv.closed_timestamp.target_duration = '1s';
SET CLUSTER SETTING kv.rangefeed.closed_timestamp_refresh_interval = '250ms';
SET CLUSTER SETTING kv.rangefeed.scheduler.enabled = true;

-- Improve throughput during backfills or after range movement. Increases cluster CPU.
SET CLUSTER SETTING kv.rangefeed.concurrent_catchup_iterators = 64; --default is 16

-- Removed in 24.1 and after.
-- https://www.cockroachlabs.com/docs/releases/v24.1#v24-1-0-alpha-1-operational-changes
SET CLUSTER SETTING kv.rangefeed.catchup_scan_concurrency = 64;  --default is 8


-- Decreases commit-to-emit latency.
SET CLUSTER SETTING changefeed.experimental_poll_interval = '10ms';

-- We can become limited on the number of concurrent HTTP requests that a
-- cockroach node will make. This is relevant when emitting large transactions.
-- Check Replicator staging performance dashboard when tuning this.
SET CLUSTER SETTING changefeed.sink_io_workers = 128;

-- Enable schema-locking on each table that is part of the changefeed.
-- This can be set and reset on the fly. It allows the changefeed emitter
-- to skip schema versioning checks in the steady-state.
ALTER TABLE foo_tbl SET(schema_locked=t);
ALTER TABLE foo_tbl RESET schema_locked; -- If/when schema changes are necessary.


-- Minimum required WITH options for webhook delivery.
CREATE CHANGEFEED FOR TABLE YCSB.USERTABLE
  INTO 'webhook-https://127.0.0.1:30004/ycsb/public?insecure_tls_skip_verify=true'
  WITH updated, resolved='0', min_checkpoint_frequency='0',
       webhook_sink_config='{"Flush":{"Bytes":1048576,"Frequency":"250ms"}}';

Idempotent redelivery

CockroachDB changefeeds guarantee at-least-once delivery of each message, but do not guarantee idempotent redelivery of mutations under all circumstances in the default configuration. That is, for some row R, it would be possible to see a sequence of row timestamps (1, 2, 3, 2, 4) or just (1, 2, 3, 2).

Replicator's default, fully-consistent mode inherently debounces repeated deliveries. However, to prevent a row from being rolled back in eventually-consistent modes (BestEffort or Immediate), Replicator must write "stub" entries to the staging tables to know which versions of a row have already been applied to the target.

Idempotent changefeed redelivery can be configured by setting the changefeed.frontier_checkpoint_frequency cluster setting to 0. Once this has been done, the --assumeIdempotent flag may be passed to the start command to disable the extra version-tracking queries.

High network latency

In deployments with high network latency, consider bandwidth delay product effects and optimize for larger transfer windows.

Pre-splitting

Pre-splitting tables across a larger number of ranges also provides additional opportunities for concurrent data transfer between CockroachDB and Replicator.

Additional guidance can be found at CRDB Advanced Chagnefeed Configuration

Workload generator

See here.

Discard mode

The replicator start command supports a --discard flag that will discard incoming changefeed payloads and immediately return a 200 OK response. This mode can be used to test the maximum throughput that any particular source cluster and Replicator deployment can achieve. An accompanying --discardDelay flag can add an artificial delay to this mode to help with BDP-related testing.