Skip to content

Troubleshooting

Jump to bottom
Bob Vawter edited this page Jan 19, 2024 · 19 revisions

Troubleshooting

Questions

  • Are there errors in the cdc-sink logs?
    • Using --logFormat fluent and --logDestination path/to/cdc-sink.log are a good choice for setting up log aggregation, especially if cdc-sink is being run as a replicated network service.
  • Is the source changefeed able to deliver data to cdc-sink?
    • Check the output of SHOW CHANGEFEED JOBS
    • Are there (retryable) error messages reported by the changefeed senders?
    • Is the resolved timestamp advancing?
    • Are webhook flush options set to enable bulk delivery?
  • Is cdc-sink actively staging data?
    • Look at the row counts for the staging tables in the _cdc_sink database.
  • Is cdc-sink receiving resolved timestamps?
    • Look at the row count of _cdc_sink.resolved_timestamps
    • Performing a transactionally-consistent backfill is not recommended, since this would require the initial state of the database to be applied in a single transaction. Use --immediate or --backfillWindow modes, or bootstrap the destination from a BACKUP or EXPORT.
  • Are resolved timestamps falling behind?
    • SELECT now() - MAX(target_applied_at) FROM _cdc_sink.resolved_timestamps to show when a resolved window was last processed.

Actions

Internal diagnostic endpoint

cdc-sink provides introspection of its internal datastructures though a diagnostic endpoint at /_/diag. This endpoint will return a JSON blob describing many of cdc-sink's internal datastructure. The payload may contain sensitive information. If cdc-sink authentication is enabled, the requestor must have access permissions to a schema named _.diag in order to make the request.

This same information can be sent to cdc-sink's logger by sending a SIGUSR1 to the cdc-sink process.

This data is for support purposes only and does not constitute a stable API.

Reset cdc-sink

  • Cancel all source changefeeds
  • DROP and re-CREATE the _cdc_sink database.

Decode memo table entries

The information in the _cdc_sink.memo table is typically available from the diagnostic bundle. The data in the table can be decoded using SELECT * FROM memo WHERE key LIKE 'changefeed-%' and decode the value using pbpaste | xxd -r -p -.

Override apply templates

The SQL queries in the apply package are generated from golang templates embedded into the binary. It's possible load the templates from disk for temporary troubleshooting or experimentation. The templates are tightly coupled with internal data structures, so we cannot provide any long-term guarantees about compatibility.

# Write templates to the current directory
cdc-sink dumptemplates --path .
# Edit ./queries/<db>/query.tmpl
CDC_SINK_TEMPLATES=. cdc-sink start ...
Clone this wiki locally