bug: unable to complete cuda prover on a small blcok

[gbotrel]

What I run on my fork (i.e. #22 ):

RUST_BACKTRACE=1 cargo run --bin rsp --release --features cuda  -- --block-number 2 --rpc-url ... --prove

CUDA prover always panic at some point (instance is aws ec2 g6.2x w/ cuda 12 + cuda toolkit etc).

2024-08-28T20:36:04.385151Z  INFO prove_core: Chip Byte                : 28639232             =        437W x      65536H
2024-08-28T20:36:04.385153Z  INFO prove_core: Chip ShiftLeft           : 1998848              =         61W x      32768H
2024-08-28T20:36:04.385155Z  INFO prove_core: Chip ShiftRight          : 2080768              =        127W x      16384H
thread 'main' panicked at /home/ec2-user/.cargo/git/checkouts/sp1-20c98843a1ffc860/fb967e8/crates/cuda/src/lib.rs:197:82:
called Result::unwrap() on an Err value: ReqwestError(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("localhost")), port: Some(3000), path: "/twirp/api.ProverService/ProveCore", query: None, fragment: None }, source: hyper_util::client::legacy::Error(SendRequest, hyper::Error(IncompleteMessage)) })
stack backtrace:
   0: rust_begin_unwind
   1: core::panicking::panic_fmt
   2: core::result::unwrap_failed
   3: sp1_cuda::SP1CudaProver::prove_core
   4: <sp1_sdk::provers::cuda::CudaProver as sp1_sdk::provers::Prover<sp1_prover::components::DefaultProverComponents>>::prove
   5: sp1_sdk::action::Prove::run
   6: tokio::runtime::park::CachedParkThread::block_on
   7: tokio::runtime::runtime::Runtime::block_on
   8: rsp::main

[jtguibas]

@gbotrel Let me know if it works for you now and we can close this issue!

[xJonathanLEI]

@jtguibas Is there a reason this is re-opened? It looks like the prover server gets OOM killed or something here.

Should we add CUDA CI here?