RPCDaemon and Erigon: don't open files at startup if download is not finished (StageSnapshots == 0) #12332
Conversation
AskAlexSharov
changed the title
AskAlexSharov
changed the title
cmd/rpcdaemon/cli/config.go
Outdated
| allBorSnapshots.OptimisticalyReopenFolder() | ||
| if doOptimisticOpen { | ||
| allSnapshots.OptimisticalyReopenFolder() | ||
| allBorSnapshots.OptimisticalyReopenFolder() |
There was a problem hiding this comment.
the naming is OptimisticallReopenFolder, but the ReopenFolder function it called doesn't open file in optimistic way.
AskAlexSharov
changed the title
erigon-lib/downloader/downloader.go
Outdated
| @@ -2319,6 +2293,10 @@ func (d *Downloader) ReCalcStats(interval time.Duration) { | |||
|
|
|||
| d.lock.Unlock() | |||
|
|
|||
| if stats.Completed { | |||
| d.saveAllCompleteFlag() | |||
There was a problem hiding this comment.
There're two steps of downloading: 1. header chain segments(headers and bodies) 2. other segments(transactions...)
stats.Completed is true after the first step completed, but will be re-written to false once step2 starts.
should we wait all types of segments complete and then set the flag?
There was a problem hiding this comment.
add some logs locally and here's logs test on sepolia:
Stats.Completed false
[INFO] [10-17|23:03:55.301] [1/6 OtterSync] Downloading progress="(228/228 files) 99.71% - 2.7GB/2.7GB" rate=5.9MB/s time-left=0hrs:0m total-time=1m40s download-rate=5.9MB/s completion-rate=5.9MB/s alloc=552.7MB sys=653.8MB
Stats.Completed true
header and body snapshots downloaded completely
other snapshots downloading started
[INFO] [10-17|23:03:55.810] [snapshots] no metadata yet files=90 list=domain/v1-accounts.0-128.bt,domain/v1-receipt.200-202.kv,v1-006710-006720-transactions.idx,v1-006730-006740-transactions.seg,domain/v1-accounts.192-200.bt,...
Stats.Completed false
There was a problem hiding this comment.
And i'm a little confused that the flag seems to work the same way as prohibit_new_downloads.lock file, cause the file is written in right after specific types of segments complete downloading and stats.Completed turns true. However you said In the past i used prohibit_new_downloads.lock file for this - but now it created before download completion (and it’s fine).
If the prohibit_new_downloads file cannot mark the completion of download, then the flag cannot too.
If above is right, we can still use the way used by previous version of the PR , this function should work:
e729c1f#diff-f98a3c65d957c3d92bee0d12aeed5ec1ee902de014f1cda287f430808ea1aa2eR63
There was a problem hiding this comment.
You right. Maybe i must use “StageSnapshots progress > 0”
AskAlexSharov
changed the title
|
step2: #12362 |
only reason why we did it - is because we opened them first time when downloading is not done yet. but after #12332 - it can't happen anymore
AskAlexSharov
changed the title
Probably right way is: check
StageSnapshots > 0as "all downloads done". Because:preverified.tomlstays the samepreverified.tomlhas more filespreverified.tomlstays the samepreverified.tomlstays the sameCompleteflag can't be used - because e3 does download headers and bodies first, then Complete=true happens, and then we download transactions (Complete=false happens).