Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tap Postgres FastSync inconsistent snapshot for logical replication #930

Open
judahrand opened this issue Apr 2, 2022 · 0 comments · May be fixed by #929
Open

Tap Postgres FastSync inconsistent snapshot for logical replication #930

judahrand opened this issue Apr 2, 2022 · 0 comments · May be fixed by #929
Labels
bug Something isn't working

Comments

@judahrand
Copy link
Contributor

judahrand commented Apr 2, 2022

Describe the bug

I believe there is an edge case in the initial snapshot process for Postgres LOG_BASED replication.

This can only occur when snapshotting from a 'replica' and when the replication slot did not previously exist.

Under the above conditions it is possible for the following to occur:

  1. a replication slot is created on the primary database
  2. the most recent LSN of present on the replica is used as the starting point for logical replication
  3. this LSN is actually before the replication slot was created / became consistent due to replication lag between the primary and replica
  4. the replica is snapshotted but has missing data that will never be available to the replication slot

I believe this can be solved by #929. This approach is inspired heavily by how Debezium does its initial snapshotting with the addition of waiting for the replica to catch up enough with the primary if needed.

To Reproduce
Steps to reproduce the behavior:

  1. Difficult

Expected behavior
I expect all rows in the source database to be present in the destination database.

Screenshots
If applicable, add screenshots to help explain your problem.

Your environment

  • Version of PipelineWise, e.g branch/commit #/release/tag
  • Source type and setup
  • Target type and setup

Additional context
Add any other context about the problem here.

@judahrand judahrand added the bug Something isn't working label Apr 2, 2022
@judahrand judahrand linked a pull request Apr 4, 2022 that will close this issue
14 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant