Skip to content

Troubleshooting

Jump to bottom
Bob Vawter edited this page Sep 5, 2023 · 19 revisions

Troubleshooting

Questions

  • Are there errors in the cdc-sink logs?
    • Using --logFormat fluent and --logDestination path/to/cdc-sink.log are a good choice for setting up log aggregation, especially if cdc-sink is being run as a replicated network service.
  • Is the source changefeed able to deliver data to cdc-sink?
    • Check the output of SHOW CHANGEFEED JOBS
    • Are there (retryable) error messages reported by the changefeed senders?
    • Is the resolved timestamp advancing?
  • Is cdc-sink actively staging data?
    • Look at the row counts for the staging tables in the _cdc_sink database.
  • Is cdc-sink receiving resolved timestamps?
    • Look at the row count of _cdc_sink.resolved_timestamps
    • Performing a transactionally-consistent backfill is not recommended, since this would require the initial state of the database to be applied in a single transaction. Use --immediate or --backfillWindow modes, or bootstrap the destination from a BACKUP or EXPORT.
  • Are resolved timestamps falling behind?
    • SELECT now() - MAX(target_applied_at) FROM _cdc_sink.resolved_timestamps to show when a resolved window was last processed.

Actions

Internal diagnostic endpoint

cdc-sink provides introspection of its internal datastructures though a diagnostic endpoint at /_/diag. This endpoint will return a JSON blob describing many of cdc-sink's internal datastructure. The payload may contain sensitive information. If cdc-sink authentication is enabled, the requestor must have access permissions to a schema named _.diag in order to make the request.

This same information can be sent to cdc-sink's logger by sending a SIGUSR1 to the cdc-sink process.

This data is for support purposes only and does not constitute a stable API.

Reset cdc-sink

  • Cancel all source changefeeds
  • DROP and re-CREATE the _cdc_sink database.

Decode memo table entries

The information in the _cdc_sink.memo table is typically available from the diagnostic bundle. The data in the table can be decoded using SELECT * FROM memo WHERE key LIKE 'changefeed-%' and decode the value using pbpaste | xxd -r -p -.

Clone this wiki locally