Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Don't re-open frozen segments #12299

Closed
mh0lt opened this issue Oct 14, 2024 · 2 comments
Closed

Don't re-open frozen segments #12299

mh0lt opened this issue Oct 14, 2024 · 2 comments
Labels
erigon3 imp2 Medium importance
Milestone

Comments

@mh0lt
Copy link
Contributor

mh0lt commented Oct 14, 2024

At the moment

func (s *RoSnapshots) rebuildSegments(fileNames []string, open bool, optimistic bool) error

Reopens all segments even if they are already open and frozen, hence will not change due to extract and merge operations.

This is inefficient as it closes and open files which costs os resources and potentially disturbs file caching.

@mh0lt mh0lt added imp2 Medium importance erigon3 labels Oct 14, 2024
@mh0lt mh0lt added this to the 3.0.0-beta1 milestone Oct 14, 2024
@stevemilk
Copy link
Contributor

stevemilk commented Oct 14, 2024

During snapshotsync, part of downloaded files can be frozen, so we need to reopen in case files are partially downloaded. Hence I think question goes back to how to verify the integrity of files.
It occurred to me that we can add a flag: downloadComplete and set it to be true after snapshotSync complete downloading so that subsequent reopen will not open those opened fils. The only flaw is it has to wait all files complete downloading.
I draft a PR and you can check if it's viable #12307.

btw, I prepared #12245 for this kind of issue too, need to pass the integration tests first though(i'm following up). This intends to maintain commitment files for downloaded files.
hope to get your advice, thank you :)

@AskAlexSharov
Copy link
Collaborator

done by #12332

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
erigon3 imp2 Medium importance
Projects
None yet
Development

No branches or pull requests

4 participants