[wip] Partial unfold during commitment #11350
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This reverts commit 62ed361.
awskii
changed the title
|
so i figured out account reducing but did not fully figured storage hash reuse |
|
working version for account 141d862 |
Added debug env `KV_READ_METRICS=false`. When it's true, we gather
Summaries of:
```
kv_get{level="L0"`}
kv_get{level="L1"`}
kv_get{level="L2"`}
kv_get{level="L3"`}
kv_get{level="L4"`}
kv_get{level="recent"}
its a first step to #10691.
Have to add per-domain metrics as well.
I will cover grafana dashboard a bit later.
|
latest working f226ae6, merge main breaks existing node commitmetn |
Giulio2002
added a commit
that referenced
this pull request
Oct 21, 2024
take2 on #11536 #11350 Partial unfold means that we want to skip unfolding of nodes when it's possible to do so. Node state for account represented by fields [balance/nonce/codeHash] and for storage by it's value. By default node is not loaded with state. During folding, if needed, state parts are loaded. Skip is possible in following cases (having memoised state hash already exists): - no changes to node state (balance/nonce/code/storage was not changed) - node preserves it's position in trie: keeps depth, keeps nibble - no subtree updates (eg. no storage updates for account or no extension split due to insertion) There is still a Q which state hashes are worth keeping and which are not. In bor experiments we've seen that top-level hashes of storages are changing quite frequently, as well as hashes of accounts with lot's of storage keys in it. This PR totally reworks mechanism of rebuilding commitment files when all other domains exists and have same ranges. - Processes big shards with smaller parts (eg 0-1024.kv is 8 shards by 128 steps) - shards become merged when range is finished (from example above, steps will be merged into 0-1024, when all 8 shards are ready (madv friendly way)) - does not write commitment into db, dumps file directly on disk If cancelled during rebuilt, have to remove all shards which doesn't have matching range to other domains. In example, `accounts` are [0-1024, 1024-1536, ..] and `commitment` are [0-1024, 1024-1152..] you have to do **remove** commitment files of steps > 1024 (their range does not match to account 1024-1536, rebuilt will be incorrect and there is no protection yet for that case. To rebuild commitment: ``` $ make integration && ./build/bin/integration commitment_rebuild --datadir <> --chain <> ``` To be done: - [x] squeezing commitment - [ ] move this subfunction into `integration commitment` along with analysis tool for commitment files. --------- Co-authored-by: alex.sharov <[email protected]> Co-authored-by: Giulio <[email protected]> Co-authored-by: Mark Holt <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
#11328