Boot stream node from local persistent stream storage #1789
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ching all streams on boot
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎ 1 Skipped Deployment
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
As part of the stream node boot cycli it reads streams from the streams registry and initialises its internal stream cache. Nodes read all streams and filters out streams it is not participating in before loading them into the stream cache. In test environments nodes are able to load ~15K streams/s.
At the moment omega has 1534869 streams. Under the assumption that performance is similar it takes over 100s just to load the streams. When the number of streams increase the time to retrieve will increase proportionally. With the number of nodes increasing the percentage of streams a node is participating in will decrease causing the node to ignore relatively more and more streams wasting resources.
This PR uses a different approach and tracks stream data from the streams registry into its local database. On boot it fetches the streams from the database and applies changes. Fetching changes can be done by retrieving logs and is more efficient than fetching all streams. This reduces boot time and scales better over time when the number of streams and nodes increases.