This repository has been archived by the owner on May 30, 2022. It is now read-only.
Reduce memory footprint due to storing state messages #2
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This only documents `tests/unit` as `tests/migrations` appear to be designed to work inside a container rather than a local dev machine.
ericboucher
approved these changes
Feb 9, 2022
There was a problem hiding this comment.
Looks good, I'm just wondering if it might make sense to propose this as a config? Or at least code with that in mind?
So for example, handle_state_message could take in both line and line_data as arguments and then decide what to do with it based on the config?
25 tasks
…bution Adds contribution docs to understand how to run the tests
…tamill-co#200) Co-authored-by: Eric Boucher <[email protected]>
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR helps substantially reduce the memory footprint of the target by storing the singer STATE message in their raw string format instead of the deserialized objects.
The downside is that STATE messages are deserialized twice instead of just one time, so there is some slight performance penalty. However, this only really applies to 1/N such messages, with N depending on the
max__batch_rowsconfig setting, but typically appearing to be >100.In practice, we've seen a memory footprint reduction by a factor of 15-20x with this change, this will of course depend on how your STATE messages are structured.