Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

freezeblocks: fix max visible segment when there is a gap in .idx files #12263

Merged
merged 1 commit into from
Oct 9, 2024

Conversation

taratorio
Copy link
Member

@taratorio taratorio commented Oct 9, 2024

We have a gap in our bor mainnet snapshot toml between
'v1-051900-052000-headers.idx' = 'd97b3c598f520888eb67244ba28d994998584e65'
'v1-054000-054100-headers.idx' = 'a20efd5aca03ebb528215b24ce6aa0c20cc90c55'

When syncing bor mainnet from scratch the node fails to sync due to that. We will fix the gap in our toml however the node should auto-recover by regenerating the .idx files

After digging further it seems like we have a regression in #11557
idxAvailability used to stop when finding a seg which is not indexed
after that PR it does not, but instead it keeps on going so idxMax gets set to 62 mil although there is a gap (it should have been set to 52 mil where the gap is)

Tested on my node and it recreated the gap of indices

[INFO] [10-09|16:32:27.574] [snapshots:missed-idx] Stat              blocks=62.44M indices=52.00M alloc=26.8GB sys=27.4GB
[DBUG] [10-09|16:32:27.957] [index] created                          file=/home/admin/erigon-data/e3-bor-heimdall-stage-mainnet/snapshots/v1-052500-052600-headers.idx.tmp
[DBUG] [10-09|16:32:27.957] [index] calculating                      file=v1-052500-052600-headers.idx
[DBUG] [10-09|16:32:27.977] [index] created                          file=/home/admin/erigon-data/e3-bor-heimdall-stage-mainnet/snapshots/v1-053200-053300-headers.idx.tmp
[DBUG] [10-09|16:32:27.977] [index] calculating                      file=v1-053200-053300-headers.idx
[DBUG] [10-09|16:32:27.978] [index] created                          file=/home/admin/erigon-data/e3-bor-heimdall-stage-mainnet/snapshots/v1-052800-052900-headers.idx.tmp
[DBUG] [10-09|16:32:27.978] [index] calculating                      file=v1-052800-052900-headers.idx
[DBUG] [10-09|16:32:27.978] [index] created                          file=/home/admin/erigon-data/e3-bor-heimdall-stage-mainnet/snapshots/v1-053000-053100-headers.idx.tmp
[DBUG] [10-09|16:32:27.978] [index] calculating                      file=v1-053000-053100-headers.idx
[DBUG] [10-09|16:32:27.980] [index] created                          file=/home/admin/erigon-data/e3-bor-heimdall-stage-mainnet/snapshots/v1-052300-052400-headers.idx.tmp
[DBUG] [10-09|16:32:27.980] [index] calculating                      file=v1-052300-052400-headers.idx
[DBUG] [10-09|16:32:27.983] [index] created                          file=/home/admin/erigon-data/e3-bor-heimdall-stage-mainnet/snapshots/v1-052700-052800-headers.idx.tmp
[DBUG] [10-09|16:32:27.983] [index] calculating                      file=v1-052700-052800-headers.idx
[DBUG] [10-09|16:32:27.990] [index] created                          file=/home/admin/erigon-data/e3-bor-heimdall-stage-mainnet/snapshots/v1-052400-052500-headers.idx.tmp
[DBUG] [10-09|16:32:27.990] [index] calculating                      file=v1-052400-052500-headers.idx
[DBUG] [10-09|16:32:27.994] [index] created                          file=/home/admin/erigon-data/e3-bor-heimdall-stage-mainnet/snapshots/v1-053300-053400-headers.idx.tmp
[DBUG] [10-09|16:32:27.994] [index] calculating                      file=v1-053300-053400-headers.idx
[DBUG] [10-09|16:32:27.996] [index] created                          file=/home/admin/erigon-data/e3-bor-heimdall-stage-mainnet/snapshots/v1-052200-052300-headers.idx.tmp
[DBUG] [10-09|16:32:27.996] [index] calculating                      file=v1-052200-052300-headers.idx
[DBUG] [10-09|16:32:27.997] [index] created                          file=/home/admin/erigon-data/e3-bor-heimdall-stage-mainnet/snapshots/v1-052600-052700-headers.idx.tmp
[DBUG] [10-09|16:32:27.997] [index] calculating                      file=v1-052600-052700-headers.idx
[DBUG] [10-09|16:32:28.001] [index] created                          file=/home/admin/erigon-data/e3-bor-heimdall-stage-mainnet/snapshots/v1-052100-052200-headers.idx.tmp
[DBUG] [10-09|16:32:28.001] [index] calculating                      file=v1-052100-052200-headers.idx
[DBUG] [10-09|16:32:28.002] [index] created                          file=/home/admin/erigon-data/e3-bor-heimdall-stage-mainnet/snapshots/v1-052000-052100-headers.idx.tmp
[DBUG] [10-09|16:32:28.002] [index] calculating                      file=v1-052000-052100-headers.idx
[DBUG] [10-09|16:32:28.016] [index] created                          file=/home/admin/erigon-data/e3-bor-heimdall-stage-mainnet/snapshots/v1-053100-053200-headers.idx.tmp
[DBUG] [10-09|16:32:28.016] [index] calculating                      file=v1-053100-053200-headers.idx
[DBUG] [10-09|16:32:28.020] [index] created                          file=/home/admin/erigon-data/e3-bor-heimdall-stage-mainnet/snapshots/v1-052900-053000-headers.idx.tmp
[DBUG] [10-09|16:32:28.020] [index] calculating                      file=v1-052900-053000-headers.idx
[DBUG] [10-09|16:32:28.023] [index] created                          file=/home/admin/erigon-data/e3-bor-heimdall-stage-mainnet/snapshots/v1-053400-053500-headers.idx.tmp
[DBUG] [10-09|16:32:28.023] [index] calculating                      file=v1-053400-053500-headers.idx
[DBUG] [10-09|16:32:28.235] [index] write                            file=v1-053200-053300-headers.idx
[DBUG] [10-09|16:32:28.237] [index] write                            file=v1-052800-052900-headers.idx
[DBUG] [10-09|16:32:28.237] [index] write                            file=v1-052700-052800-headers.idx
[DBUG] [10-09|16:32:28.244] [index] write                            file=v1-052400-052500-headers.idx
[DBUG] [10-09|16:32:28.244] [index] write                            file=v1-052100-052200-headers.idx
[DBUG] [10-09|16:32:28.265] [index] write                            file=v1-053000-053100-headers.idx
[DBUG] [10-09|16:32:28.268] [index] write                            file=v1-052300-052400-headers.idx
[DBUG] [10-09|16:32:28.270] [index] write                            file=v1-052500-052600-headers.idx
[DBUG] [10-09|16:32:28.286] [index] write                            file=v1-052900-053000-headers.idx
[DBUG] [10-09|16:32:28.288] [index] write                            file=v1-052600-052700-headers.idx
[DBUG] [10-09|16:32:28.292] [index] write                            file=v1-052000-052100-headers.idx
[DBUG] [10-09|16:32:28.296] [index] write                            file=v1-053100-053200-headers.idx
[DBUG] [10-09|16:32:28.319] [index] write                            file=v1-052200-052300-headers.idx
[DBUG] [10-09|16:32:28.322] [index] write                            file=v1-053300-053400-headers.idx
[DBUG] [10-09|16:32:28.339] [index] write                            file=v1-053400-053500-headers.idx
[DBUG] [10-09|16:32:28.508] [index] created                          file=/home/admin/erigon-data/e3-bor-heimdall-stage-mainnet/snapshots/v1-053800-053900-headers.idx.tmp
[DBUG] [10-09|16:32:28.508] [index] calculating                      file=v1-053800-053900-headers.idx
[DBUG] [10-09|16:32:28.520] [index] created                          file=/home/admin/erigon-data/e3-bor-heimdall-stage-mainnet/snapshots/v1-053900-054000-headers.idx.tmp
[DBUG] [10-09|16:32:28.520] [index] calculating                      file=v1-053900-054000-headers.idx
[DBUG] [10-09|16:32:28.521] [index] created                          file=/home/admin/erigon-data/e3-bor-heimdall-stage-mainnet/snapshots/v1-053600-053700-headers.idx.tmp
[DBUG] [10-09|16:32:28.521] [index] calculating                      file=v1-053600-053700-headers.idx
[DBUG] [10-09|16:32:28.526] [index] created                          file=/home/admin/erigon-data/e3-bor-heimdall-stage-mainnet/snapshots/v1-053700-053800-headers.idx.tmp
[DBUG] [10-09|16:32:28.526] [index] calculating                      file=v1-053700-053800-headers.idx
[DBUG] [10-09|16:32:28.529] [index] created                          file=/home/admin/erigon-data/e3-bor-heimdall-stage-mainnet/snapshots/v1-053500-053600-headers.idx.tmp
[DBUG] [10-09|16:32:28.529] [index] calculating                      file=v1-053500-053600-headers.idx
[DBUG] [10-09|16:32:28.756] [index] write                            file=v1-053800-053900-headers.idx
[DBUG] [10-09|16:32:28.780] [index] write                            file=v1-053900-054000-headers.idx
[DBUG] [10-09|16:32:28.782] [index] write                            file=v1-053600-053700-headers.idx
[DBUG] [10-09|16:32:28.783] [index] write                            file=v1-053700-053800-headers.idx
[DBUG] [10-09|16:32:28.805] [index] write                            file=v1-053500-053600-headers.idx
[INFO] [10-09|16:32:30.944] [snapshots:missed-idx:reopen] Stat       blocks=62.44M indices=62.44M alloc=21.7GB sys=32.5GB

@AskAlexSharov AskAlexSharov enabled auto-merge (squash) October 9, 2024 16:53
@AskAlexSharov AskAlexSharov merged commit 05fb023 into main Oct 9, 2024
11 checks passed
@AskAlexSharov AskAlexSharov deleted the fix-visible-segments-when-idx-gap branch October 9, 2024 17:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants