Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make Flows crash resistant #183

Open
rpanic opened this issue Jul 25, 2024 · 0 comments
Open

Make Flows crash resistant #183

rpanic opened this issue Jul 25, 2024 · 0 comments

Comments

@rpanic
Copy link
Member

rpanic commented Jul 25, 2024

Is your feature request related to a problem? Please describe.
Currently, if the sequencer crashes during batch production, it has to restart the whole process again, since it has no knowledge of intermediate results.
The queue can't do that, since the sequencer needs to consume task results eagerly in order to create new merging tasks (or other tasks continuing the flow, i.e. runtime -> tx proofs -> block proofs).
Therefore we need to make sure the sequencer has a way to recover the already created or completed tasks to continue block production where it left off.

Describe the solution you'd like
The Flow utilities should accept some module that allows for it to save dispatches tasks and, when completed, their results. Identification happens based on a hash of the task type plus all inputs. Therefore, any hash equality means that the inputs are the same, therefore the result will also always be the same => stored result can be reused

Open questions:

  • Merge pairing happens eagerly and is therefore non-deterministic. That could lead to the cache not hitting
    Example: Tasks A - B - C should be merged. The Flow framework could merge B - C => BC first, while on the restarted flow, A - B => AB could be paired first, leading to a cache miss.
@rpanic rpanic moved this to Backlog in Main Board Jul 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Backlog
Development

No branches or pull requests

1 participant