FastSync with tap_postgres fails to set last_replication_method key in state bookmarks

[rscheuermann]

Describe the bug
After running a complete tap (tap-postgres to target-snowflake) using only FastSync for the initial load, my state file contains bookmarks that look like this:

"public-file_widgets": {
      "lsn": 129769327017984,
      "version": 1
    }

Once complete, and I try to run the tap using the singer method, I get an error:

Traceback (most recent call last):
  File "/app/.virtualenvs/tap-postgres/bin/tap-postgres", line 8, in <module>
    sys.exit(main())
  File "/app/.virtualenvs/tap-postgres/lib/python3.7/site-packages/tap_postgres/__init__.py", line 434, in main
    raise exc
  File "/app/.virtualenvs/tap-postgres/lib/python3.7/site-packages/tap_postgres/__init__.py", line 431, in main
    main_impl()
  File "/app/.virtualenvs/tap-postgres/lib/python3.7/site-packages/tap_postgres/__init__.py", line 421, in main_impl
    args.config.get('default_replication_method'), state, state_file)
  File "/app/.virtualenvs/tap-postgres/lib/python3.7/site-packages/tap_postgres/__init__.py", line 321, in do_sync
    state = sync_logical_streams(conn_config, list(streams), state, end_lsn, state_file)
  File "/app/.virtualenvs/tap-postgres/lib/python3.7/site-packages/tap_postgres/__init__.py", line 216, in sync_logical_streams
    if bookmark == {} or bookmark['last_replication_method'] != 'LOG_BASED' or stream in selected_streams:
KeyError: 'last_replication_method'

The code is looking for last_replication_method on a bookmark object, but fastsync isn't setting it. See:

pipelinewise/pipelinewise/fastsync/commons/tap_postgres.py

Lines 214 to 217 in 0fa08e7

	return {
	'lsn': lsn,
	'version': 1
	}

To resolve, I added last_replication_method manually to my state file.

[Samira-El]

Hey Ryan,

Is this still happening? If no, then can I ask you to close this issue, otherwise, please elaborate a bit more on:

using only FastSync

How did you trigger fastSync? was it via the pipelinewise sync_tables command or running the postgres-to-snowflake cli?

public-file_widgets

What replication method have you configured for this table?

[rscheuermann]

I'm no longer able to replicate this issue.

[halilduygulu]

this is still like that in master https://github.com/transferwise/pipelinewise/blob/master/pipelinewise/fastsync/commons/tap_postgres.py#L327

and I am getting error in second run when singer tries to read state
#990