Proposal for block header lookup via block number in the history network #334

KolbyML · 2024-09-06T15:12:52Z

Offshoot of this PR #333 you can read for a deeper background or reasoning.

In the other PR I list Pro's and Con's of certain block index model's and in the end come to the conclusion that it would probably just make more sense to store these values in the History network, as the total storage size for these indexes will be around 22ish GB with proofs. Which is extremely small in comparison with the requirements to store block bodies and receipts which is around 749.94GB so I think it is well justified.

In the same note we have previously discussed #268 removing epoch accumulator history network content type, but decided not to due to not having a better solution during the time.

With this proposal we add block number -> block header lookup capabilities

Here is a copy paste from my other PR why we want these indexes

Most JSON-RPC calls support requesting data by block number
ETH Call requires block number -> block hash indexes for the last 256 blocks (in Portal's case we would need access to all the blocks 0 to latest to properly support eth call)
Being able to fetch header's by block number is more intuitive and easier to get what you want, our current model requires you to start from the latest block header from the beacon network and walk back to your desired block header, so potentially if you wanted the block header for block 1 you would have to fetch 20 million headers which is a little overkill

In short this proposal offers a much better UX experience and enables us to properly implement the JSON-RPC

pipermerriam · 2024-09-06T16:21:56Z

history/history-network.md

+> **_Note:_** The `BlockHeaderProof` allows to provide headers without a proof (`None`).
+For pre-merge headers, clients SHOULD NOT accept headers without a proof
+as there is the `HistoricalHashesAccumulatorProof` solution available.
+For post-merge headers, there is currently no proof solution and clients MAY


What is our plan here? Post merge we need to prove against the beacon chain root? And we have the issue of dealing with the mutable proof issue for blocks while they are in the latest 8192 blocks?

This seems like a can that keeps getting kicked down the road which needs to get dealt with soon.

Seems like we could at least define everything pre-Capella since those proofs are all stable. Is there something stopping us from doing this?

I think that is in @kdeme and @ogenev area of research. It does seem like an important issue

Add HistoricalRootsBlockProof for merge till capella blocks #287

Add HistoricalSummariesBlockProof for headers from Capella onwards #291

I think these 2 PR made by Kim cover it, and it is a little out of context for this PR, I do see the reason it is being brought up, but I think those 2 PR's are better as they were made to address that I believe

I think this note can be removed and instead just referred to the other one, instead of repeating it.

But this is not something new for this specific PR so a discussion here around this is probably not the best place, but I'll summarize it anyhow:

There is a plan for this, it is put in following 3 PRs:

Add HistoricalRootsBlockProof for merge till capella blocks #287

Add HistoricalSummariesBlockProof for headers from Capella onwards #291

Add a suggestion on how to verify recent headers #292

pre-Capella one could be activated, it mostly depends on how ready clients are for this and if we want clients to have some better system to easily prune headers without proof first.

post-Capella ones mostly rely on beacon network being fully active (and there is the recent issue of the proof increase).

I think this note can be removed and instead just referred to the other one, instead of repeating it.

I moved things around a bit let me know if there is still an issue or if you think it could be done in a cleaner way

For post capella, couldn't we do something similar to what we do with pre-merge and prove against a fixed state root for each fork. This would allow us to fill in most of history with a few frozen hashes and would eliminate the need for clients to actively access beacon network data for older fork data.

For post capella, couldn't we do something similar to what we do with pre-merge and prove against a fixed state root for each fork. This would allow us to fill in most of history with a few frozen hashes and would eliminate the need for clients to actively access beacon network data for older fork data.

Sure, specifications do not say how one must get the HistoricalSummaries.
Clients could already bake these in at a certain timestamp. Of course they would have to regularly update this in their binary to allow for newer periods to be covered. Or, a user could perhaps manually load them via CLI through some json/yaml/ssz file that holds it. If I were to implement this I'd probably go for the latter.

kdeme · 2024-09-06T16:24:03Z

history/history-network.md

@@ -20,7 +20,7 @@ In addition, the chain history network provides individual epoch records for the
    - Ommers
    - Withdrawals
 - Receipts
- Header epoch records (pre-merge only)
+- Block header indexes


I would remove this as it's actually the block headers that are provided, but accessed by number

kdeme · 2024-09-06T16:24:48Z

history/history-network.md

@@ -261,17 +259,27 @@ content_key         = selector + SSZ.serialize(receipt_key)

 Note: The type-specific receipts encoding might be different for future receipt types, but this content encoding is agnostic to the underlying receipt encodings.

-#### Epoch Record
+#### Block Header Indexes


I would call it also here Block Header by Number (and perhaps refer to the other one as by hash)

njgheorghita · 2024-09-06T17:48:25Z

history/history-network.md

@@ -8,7 +8,7 @@ The chain history network is a [Kademlia](https://pdos.csail.mit.edu/~petar/pape

 Execution chain history data consists of historical block headers, block bodies (transactions, ommers and withdrawals) and block receipts.

-In addition, the chain history network provides individual epoch records for the full range of pre-merge blocks mined before the transition to proof of stake.
+In addition, the chain history network provides block number to historical block header lookups


nitpick period at end of sentence

kdeme · 2024-09-06T19:20:34Z

history/history-network.md

+# Content and content key
+
+block_number_key = Container(block_number: uint64)
+selector         = 0x04


What's the reasoning not to reuse the 0x03 value of Epoch Records?

Avoiding some clashing of older versions that still decode this differently and it being more clear when something sends the old 0x03 data?

Avoiding some clashing of older versions that still decode this differently and it being more clear when something sends the old 0x03 data?

Initially

What's the reasoning not to reuse the 0x03 value of Epoch Records?

I am fine reusing it, I wasn't sure if that was ok, I will update it now

I'm not sure myself / have no strong preference.

With no reuse the transition period can be somewhat more clean in terms of error logs, and there are enough numbers available.
But I don't think it's really an issue to re-use either, and the end result is more clean :)

I'm inclined towards re-using rather than leaving a gap. Not a strong feeling.

ScottyPoi

Works for me.

bhartnett · 2024-09-09T04:05:58Z

Great idea here. Currently in Fluffy, our state endpoints implementation (eth_getBalance, eth_getCode, ... etc) does two lookups in order to get the state root which is required in order to return any state data. The first lookup gets the block hash for the given block number by looking up the epoch record and then looking up the block header via block hash to get the state root. With this change we will only need one history network lookup instead of two.

ogenev

⛵

Proposal for block number indexes in the history network

04a1807

KolbyML requested review from carver, pipermerriam, kdeme, njgheorghita, acolytec3, ogenev, ScottyPoi and morph-dev September 6, 2024 15:12

KolbyML self-assigned this Sep 6, 2024

pipermerriam reviewed Sep 6, 2024

View reviewed changes

kdeme reviewed Sep 6, 2024

View reviewed changes

KolbyML mentioned this pull request Sep 6, 2024

Initial draft for canonical block index network #333

Closed

KolbyML added 2 commits September 6, 2024 10:32

fix: addressing pr concerns

fd96419

clean up ordering

f473078

KolbyML changed the title ~~Proposal for block number indexes in the history network~~ Proposal for block header lookup via block number in the history network Sep 6, 2024

njgheorghita approved these changes Sep 6, 2024

View reviewed changes

kdeme reviewed Sep 6, 2024

View reviewed changes

kdeme approved these changes Sep 6, 2024

View reviewed changes

fix: resolve more PR concerns

ecdd307

ScottyPoi approved these changes Sep 7, 2024

View reviewed changes

ogenev approved these changes Sep 9, 2024

View reviewed changes

ScottyPoi mentioned this pull request Sep 9, 2024

History: header lookup by number ethereumjs/ultralight#622

Merged

KolbyML merged commit d3b92d0 into ethereum:master Sep 10, 2024
2 checks passed

KolbyML mentioned this pull request Sep 10, 2024

feat: add block by number and remove epoch accumulators ethereum/trin#1436

Merged

morph-dev mentioned this pull request Sep 12, 2024

feat: add basic support for eth_call ethereum/trin#1442

Merged

This was referenced Sep 12, 2024

Add BlockHeader by number to Portal history network status-im/nimbus-eth1#2620

Merged

Add Audit of BlockHeader by number ethereum/glados#314

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Proposal for block header lookup via block number in the history network #334

Proposal for block header lookup via block number in the history network #334

KolbyML commented Sep 6, 2024 •

edited

Loading

pipermerriam Sep 6, 2024

pipermerriam Sep 6, 2024

KolbyML Sep 6, 2024

kdeme Sep 6, 2024 •

edited

Loading

KolbyML Sep 6, 2024 •

edited

Loading

pipermerriam Sep 6, 2024

kdeme Sep 6, 2024

kdeme Sep 6, 2024

kdeme Sep 6, 2024

njgheorghita Sep 6, 2024

kdeme Sep 6, 2024 •

edited

Loading

KolbyML Sep 6, 2024

kdeme Sep 6, 2024

pipermerriam Sep 6, 2024

ScottyPoi left a comment

bhartnett commented Sep 9, 2024

ogenev left a comment

Proposal for block header lookup via block number in the history network #334

Proposal for block header lookup via block number in the history network #334

Conversation

KolbyML commented Sep 6, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kdeme Sep 6, 2024 • edited Loading

Choose a reason for hiding this comment

KolbyML Sep 6, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kdeme Sep 6, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ScottyPoi left a comment

Choose a reason for hiding this comment

bhartnett commented Sep 9, 2024

ogenev left a comment

Choose a reason for hiding this comment

KolbyML commented Sep 6, 2024 •

edited

Loading

kdeme Sep 6, 2024 •

edited

Loading

KolbyML Sep 6, 2024 •

edited

Loading

kdeme Sep 6, 2024 •

edited

Loading