Build blocks using Reth 1.1.0 #115

hashcashier · 2024-10-29T08:34:17Z

This PR migrates zeth from directly using revm to using reth for building blocks. This allows zeth to support all forks supported by reth. The changes are an overhaul of many of zeth's internal components and project structure. Notably, because of Cargo's feature unification, and reth's dependence on feature flags for enabling/disabling OP functionality, two separate zeth binaries are now necessary.

The project has been restructured into the following crates:

zeth-core Library for defining and using the sparse MPT to statelessly build a block using reth.
zeth-core-ethereum Specialization of the core library for Ethereum blocks.
zeth-core-optimism Specialization of the core library for Optimism blocks.
zeth-preflight Library for populating the data required to statelessly build a block using zeth-core.
zeth-preflight-ethereum Specialization of the preflight library for Ethereum RPCs.
zeth-preflight-optimism Specialization of the preflight library for Optimism RPCs.
zeth-guests-reth-ethereum A binary for running the stateless block building process for Ethereum blocks.
zeth-guests-reth-optimism A binary for running the stateless block building process for Optimism blocks.
zeth-guests Library for exporting the compiled rv32im bytecode of zeth-guests-reth.
zeth A library for running a CLI to use the RISC Zero zkVM to generate/verify proofs for the stateless block building process.
zeth-ethereum A binary using the zeth crate for Ethereum blocks.
zeth-optimism A binary using the zeth crate for Optimism blocks.
zeth-benchmark A binary for benchmarking proving performance for randomly sampled blocks.

Functionality changes:

All ethereum types have been migrated to alloy types.
All Bonsai and profiling logic has been delegated to r0vm's environment variables.
Optimism consensus verification (block derivation) logic has been removed.
Ethereum foundation testing is unfinished.

Note: v1.81 of the toolchain is required:

rzup update

Note: Separate binaries must be built individually

cargo build -p zeth-ethereum --bin zeth-ethereum

cargo build -p zeth-optimism --bin zeth-optimism

Issues affected:

pdg744 · 2024-11-07T22:52:42Z

@hashcashier @SchmErik iiuc v1.81 toolchain should now be ready to use!

Wollac · 2024-11-20T15:43:10Z

crates/core/src/db.rs

@@ -95,6 +95,7 @@ impl<DB: DatabaseRef + Recoverable> DatabaseRef for Wrapper<DB> {

 impl<DB: DatabaseCommit + Recoverable> DatabaseCommit for Wrapper<DB> {
    fn commit(&mut self, changes: HashMap<Address, Account>) {
+        dbg!("COMMIT!");


I believe that should be removed.

Wollac · 2024-11-20T15:50:08Z

crates/preflight/src/client.rs

+                // resolve storage orphans
+                if let Some((trie, _)) = storage_tries.get_mut(&account_proof.address) {
+                    for storage_proof in account_proof.storage_proof {
+                        // let proof_nodes = parse_proof(&storage_proof.proof)?;


There is still quite a bit of commented code, this should be removed

Wollac · 2024-11-20T15:51:49Z

crates/preflight/src/db.rs

@@ -151,6 +153,48 @@ impl<N: Network, R: CoreDriver, P: PreflightDriver<R, N>> PreflightDB<N, R, P> {
        apply_changeset(&mut self.inner, state_changeset)
    }

+    pub fn sanity_check(&mut self, state_changeset: StateChangeset) -> anyhow::Result<()> {


Unused function

Wollac · 2024-11-20T16:02:21Z

crates/preflight/src/trie.rs

-            mpt_from_proof(&proof_nodes).context("invalid storage_proof")?;
+            let proof_nodes = parse_proof(&storage_proof.proof)
+                .context("extend_proof_tries/parse storage proof")?;
+            mpt_from_proof(&proof_nodes).context(format!(


should be a with_context

Wollac · 2024-11-20T16:10:10Z

crates/preflight/src/provider/mod.rs

        Ok(vec.into_iter().collect())
    }
 }
+
+pub mod opt_ordered_map {


This is never used, ordered_map + default is used instead

Wollac · 2024-11-20T19:10:59Z

crates/core-ethereum/src/lib.rs

+
+    fn chain_spec(chain: &NamedChain) -> Option<Arc<Self::ChainSpec>> {
+        match chain {
+            NamedChain::Mainnet => Some(MAINNET.clone()),


We do not support all possible forks. I know for a fact that the pre Byzantine validation is not correct in Reth: Back then, the TxReceipt still contained intermediate state hashes which was left out in Reth for performance reasons https://github.com/paradigmxyz/reth/blob/e3702cfc87449294da061f02a571a23061b8ab50/crates/ethereum/consensus/src/validation.rs#L31
So we must exclude at least everything before Byzantine.

What about possible future forks that are (or will be with newer reth versions) only partially supported? Do they also need to be explicitly excluded?

Wollac · 2024-11-20T19:13:46Z

testing/ef-tests/testdata

I assume the submodule was removed to prevent Cargo from always checking out the huge tests.
However, then the ef-tests should be adapted to include a script or documentation that checks out the EF tests in the correct version

Wollac · 2024-11-20T19:14:57Z

Cargo.toml

@@ -1,6 +1,10 @@
 [workspace]
 resolver = "2"
-members = ["guests", "host", "lib", "primitives", "testing/ef-tests"]
+members = [


The ef-tests have been removed here. Thus, they have not been adapted to the new reth version and will not compile.

Wollac · 2024-11-20T19:17:12Z

.github/workflows/ci.yml

    - uses: risc0/cargo-install@v1
      with:
        crate: cargo-binstall
-    - run: cargo binstall -y --force cargo-risczero@$RISC0_VERSION
-    - run: cargo risczero install --version $RISC0_TOOLCHAIN_VERSION
-    - run: cargo test --workspace --all-targets -F ef-tests,debug-guest-build


We should absolutely not disable the ef-tests. Even when we are now using reth they contain crucial testcases that we must also pass.

Wollac · 2024-11-21T00:32:11Z

crates/core/src/rescue.rs

+}
+
+#[derive(Default)]
+pub struct Wrapper<T: Recoverable> {


An alternative to "abusing" the drop to get the Database back is to move the DB in and out similar to https://github.com/risc0/risc0-ethereum/blob/73eebd4efb283c7f99748d45fc730a53ee2edf5c/steel/src/contract.rs#L211
However, this should probably not also be in this PR.

hashcashier added 30 commits October 16, 2024 07:10

update zkvm and add basic reth guest

2c60f0e

remove OP derivation code

e70f1fa

switch to tiny-keccak

f6ab9e9

remove more op derive code

1e632ce

consensus checks

56fc8c8

(unfinished) refactor into core crate

6c3bfa1

post-exec split

9067612

finalization

ff8a7ea

StatelessClientStrategy

b9381f4

reth guest

90fa5e4

preflight providers

f695193

preflight db

d7a77b7

preflight client

eba328b

preflight client lib

ac54f34

uncle blocks

5688b05

reth preflight client

ac25ef5

basic bin lib

313404c

rescue db post preflight

b2de53c

provider exec fix

17a693d

refactor & fix

811dd96

block building

225b76d

zkvm exec

8bf41f4

time & ordering fix + rpc data

9464cef

proof generation

390fcf5

add prover opts to receipt file name derivation

4bbf416

receipt verification

401a061

cleanup: delete old code

0b3af8b

downgrade reth msrv + cleanup

bd2e934

block batch journal derivation

da2a99a

GAT in strategies

b80ef64

hashcashier added 21 commits October 30, 2024 15:14

simplify strategy IO

157f419

op-reth guest

64c6fca

op zeth

b87e71f

refactor

93276d8

feature unification workaround

d1d4711

op support

337e971

rename eth guest

0c9587f

fix code drop from account

a361d89

multichain support

705217c

readme

5ee963d

preflight speedup

3e5014e

clipp

8b2e87c

cache optimizations

8b42557

fix alloy primitives

2f02b37

remove ef tests submodule

db833a0

fix ci

7a38b7d

versioning nit

00c716a

ci test commands

3caf21a

fix cli short arg clash

736909a

profile flag

cb05478

benchmark tool + refactor

0583389

hashcashier added 6 commits November 11, 2024 10:48

risc0 1.1.3 w/ release builds

67a461f

fmt benchmark

2bcf20f

ci nit

496e671

ci nit

ff833d0

refactor

5bd18b7

preflight using debug namespace

0836298

Wollac reviewed Nov 21, 2024

View reviewed changes

austinabell mentioned this pull request Nov 25, 2024

switch to 1.2 rc and new k256 patch #119

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build blocks using Reth 1.1.0 #115

Build blocks using Reth 1.1.0 #115

hashcashier commented Oct 29, 2024 •

edited

Loading

pdg744 commented Nov 7, 2024

Wollac Nov 20, 2024

Wollac Nov 20, 2024

Wollac Nov 20, 2024

Wollac Nov 20, 2024

Wollac Nov 20, 2024

Wollac Nov 20, 2024

Wollac Nov 20, 2024

Wollac Nov 20, 2024

Wollac Nov 20, 2024

Wollac Nov 20, 2024

Wollac Nov 21, 2024

Build blocks using Reth 1.1.0 #115

Are you sure you want to change the base?

Build blocks using Reth 1.1.0 #115

Conversation

hashcashier commented Oct 29, 2024 • edited Loading

pdg744 commented Nov 7, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hashcashier commented Oct 29, 2024 •

edited

Loading