Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[e3] BlockReader HeaderByHash panic on sync from scratch #12069

Closed
taratorio opened this issue Sep 23, 2024 · 3 comments
Closed

[e3] BlockReader HeaderByHash panic on sync from scratch #12069

taratorio opened this issue Sep 23, 2024 · 3 comments
Assignees
@mh0lt
Labels
erigon3 imp1 High importance polygon
Milestone
3.0.0-alpha5

Comments

@taratorio
Copy link
Member

System information

Erigon version: check commit hash

OS & Version:Linux

Commit hash: 2ed4f88

Erigon Command (with flags/config):

nohup ~/erigon/build/bin/erigon --chain=bor-mainnet --bor.heimdall=http://0.0.0.0:1317 --sync.loop.block.limit=10_000 --batchSize=2g --datadir=/erigon-data/bor-heimdall-stage-mainnet --nat=extip:54.38.13.192 --metrics=true --metrics.addr=0.0.0.0 --metrics.port=8999 --pprof=true --pprof.port=6160 --txpool.disable=true --log.console.verbosity=debug &> /erigon-data/bor-heimdall-stage-mainnet/console.log &

Consensus Layer: N/A

Consensus Layer Command (with flags/config): N/A

Chain/Network: Bor mainnet

Expected behaviour

No panics when serving getBlockHeaders66

Actual behaviour

BlockReader HeaderByHash panics

Steps to reproduce the behaviour

Run a fresh sync from scratch

Backtrace

[DBUG] [09-21|14:35:15.748] [1/9 OtterSync] DONE                     in=21.49µs
[INFO] [09-21|14:35:15.748] [2/9 Headers] Waiting for headers...     from=62100028 hash=0x97271dc4392858ca4abbb76ea2e1b51425caa975c36ae71b436ffe1498688763
[DBUG] [09-21|14:35:15.748] [2/9 Headers] Requested skeleton         from=62100027 length=192
[DBUG] [09-21|14:35:16.863] [2/9 Headers] Requested skeleton         from=62100027 length=192
[INFO] [09-21|14:35:17.245] [2/9 Headers] Processed                  highest=62100029 age=0 headers=1 in=1.496 blk/sec=0
[DBUG] [09-21|14:35:17.245] [2/9 Headers] DONE                       in=1.496204567s
[INFO] [09-21|14:35:17.245] [3/9 BorHeimdall] Processing sync events... from=62100029 to=62100029
[INFO] [09-21|14:35:17.299] [3/9 BorHeimdall] Sync events            progress=62100029 lastSpanID=9704 lastSpanID=9704 lastCheckpointId=0 lastMilestoneId=0 lastStateSyncEventID=2963389 total records=0 sync event time=1.239283ms fetch time=0s snap time=106.215062ms waypoint time=0s process time=54.427075ms
[DBUG] [09-21|14:35:17.299] [3/9 BorHeimdall] DONE                   in=54.462745ms
[DBUG] [09-21|14:35:17.299] [4/9 BlockHashes] DONE                   in=52.08µs
[DBUG] [09-21|14:35:17.319] [index] created                          file=/erigon-data/bor-heimdall-stage-mainnet/snapshots/v1-062098-062099-transactions.idx.tmp
[DBUG] [09-21|14:35:17.319] [index] calculating                      file=v1-062098-062099-transactions.idx
[DBUG] [09-21|14:35:17.320] [5/9 Bodies] DONE                        in=21.233833ms
[DBUG] [09-21|14:35:17.333] [6/9 Senders] DONE                       in=12.799711ms
[DBUG] [09-21|14:35:17.568] [index] write                            file=v1-062098-062099-transactions.idx
[DBUG] [09-21|14:35:17.572] [index] created                          file=/erigon-data/bor-heimdall-stage-mainnet/snapshots/v1-062098-062099-transactions-to-block.idx.tmp
[DBUG] [09-21|14:35:17.572] [index] calculating                      file=v1-062098-062099-transactions-to-block.idx
[INFO] [09-21|14:35:17.687] [7/9 Execution] Done Commit every block  blk=62100029 blks=1 blk/s=2.8 txs=73 tx/s=206 gas/s=69.42M buf=7.0MB/2.0GB stepsInDB=0.00 step=2925.8 alloc=22.0GB sys=58.2GB
[DBUG] [09-21|14:35:17.687] [7/9 Execution] DONE                     in=354.288716ms
[DBUG] [09-21|14:35:17.689] [8/9 TxLookup] DONE                      in=1.420933ms
[DBUG] [09-21|14:35:17.689] [9/9 Finish] DONE                        in=10.86µs
[DBUG] [09-21|14:35:17.689] RPC Daemon notified of new headers       from=62100028 to=62100030 amount=1 header sending=7.95µs
[INFO] [09-21|14:35:17.689] Timings (slower than 50ms)               Headers=1.496s BorHeimdall=54ms Execution=354ms alloc=22.1GB sys=58.2GB
[DBUG] [09-21|14:35:17.689] [9/9 Finish] Prune done                  in=6.72µs
[DBUG] [09-21|14:35:17.690] [8/9 TxLookup] Prune done                in=40.16µs
[DBUG] [09-21|14:35:17.690] [7/9 Execution] Prune done               in=692.412µs
[DBUG] [09-21|14:35:17.690] [6/9 Senders] Prune done                 in=3.27µs
[DBUG] [09-21|14:35:17.690] [5/9 Bodies] Prune done                  in=2.21µs
[DBUG] [09-21|14:35:17.690] [4/9 BlockHashes] Prune done             in=4.21µs
[DBUG] [09-21|14:35:17.690] [3/9 BorHeimdall] Prune done             in=1.57µs
[DBUG] [09-21|14:35:17.690] [2/9 Headers] Prune done                 in=900ns
[DBUG] [09-21|14:35:17.690] [snapshots] Prune Blocks                 to=62098000 limit=10
[DBUG] [09-21|14:35:17.690] [snapshots] Prune Bor Blocks             to=62098000 limit=10
[DBUG] [09-21|14:35:17.690] [1/9 OtterSync] Prune done               in=49.95µs
[DBUG] [09-21|14:35:17.759] [1/9 OtterSync] DONE                     in=21.1µs
[INFO] [09-21|14:35:17.759] [2/9 Headers] Waiting for headers...     from=62100029 hash=0x7b5228ba6f28ba567f2777064d3d2be36ac1c85a7dd604af94efd38118a68f30
[DBUG] [09-21|14:35:17.760] [2/9 Headers] Requested skeleton         from=62100028 length=192
[DBUG] [09-21|14:35:17.843] [index] write                            file=v1-062098-062099-transactions-to-block.idx
unexpected fault address 0x720d28534d4c
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0x720d28534d4c pc=0xb53acd]

goroutine 2621 gp=0xc0056e8c40 m=63 mp=0xc1ea4fb008 [running]:
runtime.throw({0x2a4d5b3?, 0x27?})
        runtime/panic.go:1023 +0x5c fp=0xc0487436d0 sp=0xc0487436a0 pc=0x46125c
runtime.sigpanic()
        runtime/signal_unix.go:895 +0x285 fp=0xc048743730 sp=0xc0487436d0 pc=0x47a065
github.com/erigontech/erigon-lib/seg.(*Getter).nextPos(0x7211e739d9d8?, 0xe0?)
        github.com/erigontech/[email protected]/seg/decompress.go:578 +0xad fp=0xc048743750 sp=0xc048743730 pc=0xb53acd
github.com/erigontech/erigon-lib/seg.(*Getter).Next(0xc0487438b0, {0xc7b582bb80, 0x0, 0x80})
        github.com/erigontech/[email protected]/seg/decompress.go:689 +0xaf fp=0xc048743820 sp=0xc048743750 pc=0xb5412f
github.com/erigontech/erigon/turbo/snapshotsync/freezeblocks.(*BlockReader).headerFromSnapshotByHash(0x72898e6c1330?, {0xcd, 0x65, 0xc4, 0x46, 0x3e, 0x2e, 0xde, 0xea, 0x14, ...}, ...)
        github.com/erigontech/erigon/turbo/snapshotsync/freezeblocks/block_reader.go:912 +0x259 fp=0xc048743970 sp=0xc048743820 pc=0x16c8899
github.com/erigontech/erigon/turbo/snapshotsync/freezeblocks.(*BlockReader).HeaderByHash(0xc00274ac80, {0xf0?, 0xc048743aa8?}, {0x72898e6c1330?, 0xc1907df580?}, {0xcd, 0x65, 0xc4, 0x46, 0x3e, ...})
        github.com/erigontech/erigon/turbo/snapshotsync/freezeblocks/block_reader.go:519 +0x1b2 fp=0xc048743a20 sp=0xc048743970 pc=0x16c53f2
github.com/erigontech/erigon/eth/protocols/eth.AnswerGetBlockHeadersQuery({0x72898fa6a410, 0xc1907df580}, 0xc7b135f400, {0x72898e6c13d0, 0xc00274ac80})
        github.com/erigontech/erigon/eth/protocols/eth/handlers.go:59 +0xa83 fp=0xc048743c30 sp=0xc048743a20 pc=0x1fb24e3
github.com/erigontech/erigon/p2p/sentry/sentry_multi_client.(*MultiClient).getBlockHeaders66.func1({0x72898fa6a410?, 0xc1907df580?})
        github.com/erigontech/erigon/p2p/sentry/sentry_multi_client/sentry_multi_client.go:493 +0x5a fp=0xc048743c80 sp=0xc048743c30 pc=0x2279e9a
github.com/erigontech/erigon-lib/kv/temporal.(*DB).View(0xc79b9cd620?, {0x32bf660?, 0xc0056ec0f0?}, 0xc205299440)
        github.com/erigontech/[email protected]/kv/temporal/kv_temporal.go:103 +0xbc fp=0xc048743ce8 sp=0xc048743c80 pc=0x13d349c
github.com/erigontech/erigon/p2p/sentry/sentry_multi_client.(*MultiClient).getBlockHeaders66(0xc000a64a50, {0x32bf660, 0xc0056ec0f0}, 0xc195a73360, {0x32df020, 0xc001a16000})
        github.com/erigontech/erigon/p2p/sentry/sentry_multi_client/sentry_multi_client.go:492 +0x1cb fp=0xc048743dd8 sp=0xc048743ce8 pc=0x2279aab
github.com/erigontech/erigon/p2p/sentry/sentry_multi_client.(*MultiClient).handleInboundMessage(0x0?, {0x32bf660?, 0xc0056ec0f0?}, 0x0?, {0x32df020?, 0xc001a16000?})
        github.com/erigontech/erigon/p2p/sentry/sentry_multi_client/sentry_multi_client.go:663 +0x5b fp=0xc048743e28 sp=0xc048743dd8 pc=0x227b39b
github.com/erigontech/erigon/p2p/sentry/sentry_multi_client.(*MultiClient).HandleInboundMessage(0xc000a64a50, {0x32bf660, 0xc0056ec0f0}, 0xc195a73360, {0x32df020, 0xc001a16000})
        github.com/erigontech/erigon/p2p/sentry/sentry_multi_client/sentry_multi_client.go:634 +0xa5 fp=0xc048743ec0 sp=0xc048743e28 pc=0x227af65
github.com/erigontech/erigon/p2p/sentry/sentry_multi_client.(*MultiClient).HandleInboundMessage-fm({0x32bf660?, 0xc0056ec0f0?}, 0xc1d7405450?, {0x32df020?, 0xc001a16000?})
        <autogenerated>:1 +0x46 fp=0xc048743f00 sp=0xc048743ec0 pc=0x227c9a6
github.com/erigontech/erigon-lib/p2p/sentry.pumpStreamLoop[...].func2()
        github.com/erigontech/[email protected]/p2p/sentry/loop.go:126 +0x136 fp=0xc048743fe0 sp=0xc048743f00 pc=0x227c596
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc048743fe8 sp=0xc048743fe0 pc=0x49c601
created by github.com/erigontech/erigon-lib/p2p/sentry.pumpStreamLoop[...] in goroutine 213
        github.com/erigontech/[email protected]/p2p/sentry/loop.go:120 +0x2a7
@taratorio taratorio added imp1 High importance erigon3 labels Sep 23, 2024
@taratorio taratorio changed the title Erigon3 BlockReader HeaderByHash panic on sync from scratch [e3] BlockReader HeaderByHash panic on sync from scratch Sep 23, 2024
@AskAlexSharov AskAlexSharov added this to the 3.0.0-alpha4 milestone Sep 24, 2024
@AskAlexSharov AskAlexSharov mentioned this issue Sep 24, 2024
Merged
AskAlexSharov added a commit that referenced this issue Sep 24, 2024
@AskAlexSharov
block files: idxAvailability must rely on visibleFiles instead of ato…
2b873ff 
…mics (#12078)

reason: visible files guarantee consistency - between file types
(because hide under same mutex). `maxVisibleBlock` atomic - doesn't give
such guaranty (mutex doesn't guard them).

for: #12069
@VBulikov VBulikov modified the milestones: 3.0.0-alpha4, 3.0.0-alpha5 Sep 27, 2024
@taratorio
Copy link
Member Author

maybe solved by #12207

@taratorio
Copy link
Member Author

Nope, still happening

@AskAlexSharov
Copy link
Collaborator

solved by #12332

@yperbasis yperbasis modified the milestones: 3.0.0-alpha6, 3.0.0-alpha5 Oct 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mh0lt mh0lt

Labels
erigon3 imp1 High importance polygon
Projects
None yet
Milestone
3.0.0-alpha5
Development

No branches or pull requests
5 participants
@VBulikov @yperbasis @AskAlexSharov @taratorio @mh0lt