Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Custom Processors] [Python] Error deserializing some transactions. #375

Open
bokobza opened this issue May 15, 2024 · 0 comments
Open

[Custom Processors] [Python] Error deserializing some transactions. #375

bokobza opened this issue May 15, 2024 · 0 comments
Labels
custom-processors Issues relating to the Custom Processors

Comments

@bokobza
Copy link

bokobza commented May 15, 2024

Description

Transaction 1023992588 on testnet is failing deserialization, causing whole batches to be skipped.

The batch skipping bit has been added in #352.

  1. This approach doesn't actually skip batches, it moves the start of the next batch by 1 and then tries again.
    For the sake of argument, let's say we have a bad transaction at version 3456 and the size of batches is 5000 consistently.
    If we're processing a batch that goes from trx 2000 to trx 6999, the process will fail and will restart again, but this time trying to process trx 2001 to trx 7000. It will then fail again and again, until we get passed trx 3456, and then the process will resume without errors.

  2. The problem with that, continuing with the example above, is that if I'm interested in a transaction at version 3000, I'm never going to see it because it'll always be in a bad batch.

Ideally, the bad transaction will not fail and will either be deserialised properly, or bad fields will be ignored.

We've had to add a "slow mode" in our code so that if we see a deserialisation failure, we restart the stream asking for 1 transaction only, until we fail again, at which point we know we've processed the actual bad transaction and we can restart the stream in full speed mode.

Repro

Run the python indexer starting a bit before transaction 1023992588.

@bokobza bokobza added the custom-processors Issues relating to the Custom Processors label May 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
custom-processors Issues relating to the Custom Processors
Projects
None yet
Development

No branches or pull requests

1 participant