snos refactor (#136)

* feat(snos_job): Dependencies updated for Snos * feat(snos_job): Non-working state but prove_block added * feat(snos_job): Added storage for snos * feat(snos_job): * feat(snos_job): Stores the CairoPie as zip * feat(snos_job): Docs * feat(snos_job): Minor stuff * feat(snos_job): First set of simple tests * feat(snos_job): Comitting the error * feat(snos_job): Fixed stuff after rebase * feat(snos_job): Lint stuff * feat(snos_job): Push the bug * feat(snos_job): Compiling gpush * feat(snos_job): Fixed stuff * feat(snos_job): Nightly toolchain for CI * feat(snos_job): Mock process snos job * feat(snos_job): * feat(snos_job): * feat(snos_job): Smol fixes from review * feat(snos_job): Unwrap killed * feat(snos_job): New SNOS structure for tests * feat(snos_job): Snos Structure for tests * feat(snos_job): Added RPC url to the config * feat(snos_job): Fixed rpc * feat : snos e2e updates * feat : e2e working now * feat : coverge tests partially fixed * feat : added anvil support & fixed all tests * feat : added cairo build in workflow * Trigger Tests * workflows updated * fix : ci tests * coverage test * refactored snos changes * fixes * more fixes * refactors * cargo fmt * linting * lint * s3 key fix * change env for anvil tests * fixes * fix operator not found * fix import * cargo fmt * clippy fix * add rust version in toolchain * fix taplo * cargo fmt * cargo lock fix * fix lint * fix snos cd * fix ci * build snos file in linter * snos setup yml * attempt 2 for snos setup * added checkout * add new job * revert * add @main * uncomment carg lint * remove common snos * Revert "remove common snos" This reverts commit c6c4a9c. * one more attempt at common snos * remove common snos 2 * undo workflow comments * remove setup snos * e2e self hosted and rust toolchain * install specific rust toolchain * revert rust install * revert rustup * ci fixes * added logs to e2e * snos rpc in secret * remove override * fix broken test * add multithread to snos test * only test 1 thing * fix test * run all tests * remove one test * revert nextest * revert to nextest * prettier --------- Co-authored-by: akhercha <[email protected]> Co-authored-by: Arun Jangra <[email protected]>
madara-alliance · Oct 5, 2024 · 705b119 · 705b119
1 parent 60768cc
commit 705b119
Show file tree

Hide file tree

Showing 108 changed files with 4,025 additions and 1,236 deletions.
diff --git a/.env.example b/.env.example
@@ -47,7 +47,7 @@ MADARA_RPC_URL=
 MEMORY_PAGES_CONTRACT_ADDRESS=
 PRIVATE_KEY=
 ETHEREUM_PRIVATE_KEY=
-STARKNET_SOLIDITY_CORE_CONTRACT_ADDRESS=
+L1_CORE_CONTRACT_ADDRESS=
 
 ##### STARKNET SETTLEMENT (L3s) #####
 STARKNET_PRIVATE_KEY=

diff --git a/.env.test b/.env.test
@@ -8,8 +8,10 @@ PORT=3000
 AWS_ACCESS_KEY_ID="AWS_ACCESS_KEY_ID"
 AWS_SECRET_ACCESS_KEY="AWS_SECRET_ACCESS_KEY"
 AWS_REGION="us-east-1"
-AWS_SNS_REGION="us-east-1"
 AWS_ENDPOINT_URL="http://localhost.localstack.cloud:4566"
+
+##### Omniqueue #####
+
 AWS_DEFAULT_REGION="localhost"
 
 ##### STORAGE #####
@@ -51,17 +53,15 @@ SHARP_PROOF_LAYOUT="small"
 
 DA_LAYER="ethereum"
 SETTLEMENT_LAYER="ethereum"
-SETTLEMENT_RPC_URL="https://eth-mainnet.public.blastapi.io"
-MADARA_RPC_URL="http://localhost:5000"
-L1_CORE_CONTRACT_ADDRESS="0xc662c410C0ECf747543f5bA90660f6ABeBD9C8c4"
-MEMORY_PAGES_CONTRACT_ADDRESS="0x47312450B3Ac8b5b8e247a6bB6d523e7605bDb60"
-PRIVATE_KEY="0xdead"
+SETTLEMENT_RPC_URL="https://eth-sepolia.public.blastapi.io"
+MADARA_RPC_URL=""
+MEMORY_PAGES_CONTRACT_ADDRESS="0x07ec0D28e50322Eb0C159B9090ecF3aeA8346DFe"
 ETHEREUM_PRIVATE_KEY="0xac0974bec39a17e36ba4a6b4d238ff944bacb478cbed5efcae784d7bf4f2ff80"
-STARKNET_SOLIDITY_CORE_CONTRACT_ADDRESS="0xc662c410C0ECf747543f5bA90660f6ABeBD9C8c4"
-DEFAULT_L1_CORE_CONTRACT_ADDRESS="0xc662c410C0ECf747543f5bA90660f6ABeBD9C8c4"
-TEST_DUMMY_CONTRACT_ADDRESS="0xE5b6F5e695BA6E4aeD92B68c4CC8Df1160D69A81"
-STARKNET_OPERATOR_ADDRESS="0x2C169DFe5fBbA12957Bdd0Ba47d9CEDbFE260CA7"
-ETHEREUM_BLAST_RPC_URL="https://eth-mainnet.public.blastapi.io"
+L1_CORE_CONTRACT_ADDRESS="0xE2Bb56ee936fd6433DC0F6e7e3b8365C906AA057"
+
+##### SNOS #####
+## This is needed right now because Madara doesn't support getProof
+RPC_FOR_SNOS=""
 
 
 ##### STARKNET SETTLEMENT TEST #####
@@ -73,3 +73,7 @@ MADARA_BINARY_PATH="/path/to/madara"
 OTEL_SERVICE_NAME="madara_orchestrator"
 OTEL_COLLECTOR_ENDPOINT=""
 TRACING_LEVEL="info"
+
+##### Tests #####
+
+STARKNET_OPERATOR_ADDRESS="0x5b98B836969A60FEC50Fa925905Dd1D382a7db43"
diff --git a/.github/workflows/coverage.yml b/.github/workflows/coverage.yml
@@ -3,7 +3,9 @@ name: Rust Test & Coverage
 on:
   workflow_call:
     secrets:
-      ETHEREUM_BLAST_RPC_URL:
+      ETHEREUM_SEPOLIA_BLAST_RPC:
+        required: true
+      RPC_FOR_SNOS:
         required: true
 
 jobs:
@@ -28,8 +30,6 @@ jobs:
     steps:
       - uses: actions/checkout@v3
 
-      # selecting a toolchain either by action or manual `rustup` calls should happen
-      # before the plugin, as the cache uses the current rustc version as its cache key
       - name: Install necessary dependencies
         run: |
           sudo apt update
@@ -39,7 +39,7 @@ jobs:
         uses: actions-rs/toolchain@v1
         with:
           profile: minimal
-          toolchain: stable
+          toolchain: nightly
           override: true
           default: true
 
@@ -49,6 +49,27 @@ jobs:
       - name: Install Foundry
         uses: foundry-rs/foundry-toolchain@v1
 
+      - name: Set up Python 3.9
+        uses: actions/setup-python@v2
+        with:
+          python-version: 3.9
+
+      - name: Display Python version
+        run: python --version
+
+      - name: Install cairo 0
+        run: |
+          # Creating venv
+          python3.9 -m venv ~/cairo_venv
+          source ~/cairo_venv/bin/activate
+
+          # Installing deps
+          sudo apt install -y libgmp3-dev
+          pip3 install ecdsa fastecdsa sympy
+
+          # Installing cairo lang
+          pip3 install cairo-lang
+
       - name: Check Anvil Installation
         run: |
           if command -v anvil &> /dev/null
@@ -97,10 +118,30 @@ jobs:
       - name: Getting neccesary files for testing
         run: |
           wget -P ./crates/prover-services/sharp-service/tests/artifacts https://madara-orchestrator-sharp-pie.s3.amazonaws.com/238996-SN.zip
+          # TODO: this can be optimised
+          cp ./crates/prover-services/sharp-service/tests/artifacts/238996-SN.zip ./crates/orchestrator/src/tests/artifacts/238996-SN.zip
+
+      - name: Build SNOS files
+        run: |
+          # Downloading all the deps
+          cargo fetch
+          # Navigate to checkouts folder
+          cd /home/runner/.cargo/git/checkouts
+          # Navigating to snos
+          cd snos-*
+          # Navigating to the build
+          cd 71bd0a1
+          # Activating the venv
+          source ~/cairo_venv/bin/activate
+          # Building the cairo lang repo requirements
+          ./scripts/setup-tests.sh
 
       - name: Run llvm-cov tests
         env:
-          ETHEREUM_BLAST_RPC_URL: ${{ secrets.ETHEREUM_BLAST_RPC_URL }}
+          SETTLEMENT_RPC_URL: ${{ secrets.ETHEREUM_SEPOLIA_BLAST_RPC }}
+          RPC_FOR_SNOS: ${{ secrets.RPC_FOR_SNOS }}
+          # the self hosted runner has a different region so we override it here
+          AWS_REGION: us-east-1
         run: RUST_LOG=debug RUST_BACKTRACE=1 cargo llvm-cov nextest --release --features testing --lcov --output-path lcov.info --test-threads=1 --workspace --exclude=e2e-tests --no-fail-fast
 
       - name: Coveralls

diff --git a/.github/workflows/e2e-test.yml b/.github/workflows/e2e-test.yml
@@ -2,10 +2,13 @@ name: E2E test
 
 on:
   workflow_call:
+    secrets:
+      RPC_FOR_SNOS:
+        required: true
 
 jobs:
   coverage:
-    runs-on: ubuntu-latest
+    runs-on: karnot-arc-runner-set
 
     services:
       localstack:
@@ -24,36 +27,78 @@ jobs:
     steps:
       - uses: actions/checkout@v3
 
-      # selecting a toolchain either by action or manual `rustup` calls should happen
-      # before the plugin, as the cache uses the current rustc version as its cache key
-      - run: rustup show
+      - name: Install necessary dependencies
+        run: |
+          sudo apt update
+          sudo apt-get install -y clang llvm libudev-dev protobuf-compiler gcc g++ build-essential libssl-dev pkg-config curl wget git libclang-dev
 
       - name: Install Rust toolchain
         uses: actions-rs/toolchain@v1
         with:
           profile: minimal
-          toolchain: stable
+          toolchain: nightly
           override: true
+          default: true
 
       - name: Rust Cache
         uses: Swatinem/rust-cache@v2
 
       - name: Install Foundry
         uses: foundry-rs/foundry-toolchain@v1
 
-      - name: Check Anvil Installation
+      - name: Set up Python 3.9
+        uses: actions/setup-python@v2
+        with:
+          python-version: 3.9
+
+      - name: Display Python version
+        run: python --version
+
+      - name: Install cairo 0
+        run: |
+          # Creating venv
+          python3.9 -m venv ~/cairo_venv
+          source ~/cairo_venv/bin/activate
+
+          # Installing deps
+          sudo apt install -y libgmp3-dev
+          pip3 install ecdsa fastecdsa sympy
+
+          # Installing cairo lang
+          pip3 install cairo-lang
+
+      - name: Check Anvil Installation & run Anvil
         run: |
           if command -v anvil &> /dev/null
           then
               echo "Anvil is installed. Version information:"
               anvil --version
+              anvil &
           else
               echo "Anvil is not installed or not in PATH"
               exit 1
           fi
 
+      - name: Build SNOS files
+        run: |
+          # Downloading all the deps
+          cargo fetch
+          # Navigate to checkouts folder
+          cd /home/runner/.cargo/git/checkouts
+          # Navigating to snos
+          cd snos-*
+          # Navigating to the build
+          cd 71bd0a1
+          # Activating the venv
+          source ~/cairo_venv/bin/activate
+          # Building the cairo lang repo requirements
+          ./scripts/setup-tests.sh
+
       - name: Run e2e test
         env:
-          ETHEREUM_BLAST_RPC_URL: ${{ secrets.ETHEREUM_BLAST_RPC_URL }}
+          SETTLEMENT_RPC_URL: ${{ secrets.ETHEREUM_SEPOLIA_BLAST_RPC }}
+          RPC_FOR_SNOS: ${{ secrets.RPC_FOR_SNOS }}
+          # the self hosted runner has a different region so we override it here
+          AWS_REGION: us-east-1
         run: |
-          RUST_LOG=trace cargo test --features testing test_orchestrator_workflow -- --nocapture
+          RUST_LOG=info  cargo test --features testing test_orchestrator_workflow -- --nocapture
diff --git a/.github/workflows/linters-cargo.yml b/.github/workflows/linters-cargo.yml
@@ -11,10 +11,44 @@ jobs:
     steps:
       - uses: actions/checkout@v3
 
-      # selecting a toolchain either by action or manual `rustup` calls should happen
-      # before the plugin, as the cache uses the current rustc version as its cache key
       - run: rustup show
 
+      - name: Set up Python 3.9
+        uses: actions/setup-python@v2
+        with:
+          python-version: 3.9
+
+      - name: Display Python version
+        run: python --version
+
+      - name: Install cairo 0
+        run: |
+          # Creating venv
+          python3.9 -m venv ~/cairo_venv
+          source ~/cairo_venv/bin/activate
+
+          # Installing deps
+          sudo apt install -y libgmp3-dev
+          pip3 install ecdsa fastecdsa sympy
+
+          # Installing cairo lang
+          pip3 install cairo-lang
+
+      - name: Build SNOS files
+        run: |
+          # Downloading all the deps
+          cargo fetch
+          # Navigate to checkouts folder
+          cd /home/runner/.cargo/git/checkouts
+          # Navigating to snos
+          cd snos-*
+          # Navigating to the build
+          cd 71bd0a1
+          # Activating the venv
+          source ~/cairo_venv/bin/activate
+          # Building the cairo lang repo requirements
+          ./scripts/setup-tests.sh
+
       - uses: Swatinem/rust-cache@v2
       - name: Format and clippy
         run: |

diff --git a/.github/workflows/rust-build.yml b/.github/workflows/rust-build.yml
@@ -11,12 +11,46 @@ jobs:
     steps:
       - uses: actions/checkout@v3
 
-      # selecting a toolchain either by action or manual `rustup` calls should happen
-      # before the plugin, as the cache uses the current rustc version as its cache key
       - run: rustup show
 
       - uses: Swatinem/rust-cache@v2
 
+      - name: Set up Python 3.9
+        uses: actions/setup-python@v2
+        with:
+          python-version: 3.9
+
+      - name: Display Python version
+        run: python --version
+
+      - name: Install cairo 0
+        run: |
+          # Creating venv
+          python3.9 -m venv ~/cairo_venv
+          source ~/cairo_venv/bin/activate
+
+          # Installing deps
+          sudo apt install -y libgmp3-dev
+          pip3 install ecdsa fastecdsa sympy
+
+          # Installing cairo lang
+          pip3 install cairo-lang
+
+      - name: Build SNOS files
+        run: |
+          # Downloading all the deps
+          cargo fetch
+          # Navigate to checkouts folder
+          cd /home/runner/.cargo/git/checkouts
+          # Navigating to snos
+          cd snos-*
+          # Navigating to the build
+          cd 71bd0a1
+          # Activating the venv
+          source ~/cairo_venv/bin/activate
+          # Building the cairo lang repo requirements
+          ./scripts/setup-tests.sh
+
       - name: Build the project
         run: |
           cargo build --release --workspace
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -6,6 +6,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 
 ## Added
 
+- `SnosJob` implementation and e2e
 - Telemetry tracing and metrics.
 - e2e flow test
 - database timestamps