object_store timeout

[rkimoakbioinformatics]

I'm trying to download a public AWS S3 / Azure Storage Blob object using object_store. I read the Q&A about anonymous S3 here, but I still have the problem. However, I can download the object on S3 using Python boto3. My code goes like the following.

let store = object_store::aws::AmazonS3Builder::new()
    .with_region("us-east-1")
    .with_bucket_name("testbucket")
    .with_skip_signature(true)
    .build()
    .unwrap();
let mut stream = store.get(&object_store::path::Path::from("testfile.txt")).await;
dbg!(&stream);

The code hangs and eventually prints out the following debug message.

 &stream = Err(
    Generic {
        store: "S3",
        source: Reqwest {
            retries: 5,
            max_retries: 10,
            elapsed: 180.620760042s,
            retry_timeout: 180s,
            source: reqwest::Error {
                kind: Request,
                url: "https://s3.us-east-1.amazonaws.com/testbucket/testfile.txt",
                source: TimedOut,
            },
        },
    },
)

Any pointers would be appreciated.

[rkimoakbioinformatics]

The store dbg! output is

AmazonS3 {
    client: S3Client {
        config: S3Config {
            region: "us-east-1",
            endpoint: None,
            bucket: "testbucket",
            bucket_endpoint: "https://s3.us-east-1.amazonaws.com/testbucket",
            credentials: TokenCredentialProvider {
                inner: InstanceCredentialProvider {
                    imdsv1_fallback: false,
                    metadata_endpoint: "http://169.254.169.254",
                },
                client: Client {
                    accepts: Accepts,
                    proxies: [
                        Proxy(
                            System(
                                {},
                            ),
                            None,
                        ),
                    ],
                    referer: true,
                    default_headers: {
                        "accept": "*/*",
                        "user-agent": "object_store/0.11.0",
                    },
                    timeout: 30s,
                },
                retry: RetryConfig {
                    backoff: BackoffConfig {
                        init_backoff: 100ms,
                        max_backoff: 15s,
                        base: 2.0,
                    },
                    max_retries: 10,
                    retry_timeout: 180s,
                },
                cache: TokenCache {
                    cache: Mutex {
                        data: None,
                    },
                    min_ttl: 300s,
                },
            },
            session_provider: None,
            retry_config: RetryConfig {
                backoff: BackoffConfig {
                    init_backoff: 100ms,
                    max_backoff: 15s,
                    base: 2.0,
                },
                max_retries: 10,
                retry_timeout: 180s,
            },
            client_options: ClientOptions {
                user_agent: None,
                content_type_map: {},
                default_content_type: None,
                default_headers: None,
                proxy_url: None,
                proxy_ca_certificate: None,
                proxy_excludes: None,
                allow_http: Parsed(
                    false,
                ),
                allow_insecure: Parsed(
                    false,
                ),
                timeout: Some(
                    Parsed(
                        30s,
                    ),
                ),
                connect_timeout: Some(
                    Parsed(
                        5s,
                    ),
                ),
                pool_idle_timeout: None,
                pool_max_idle_per_host: None,
                http2_keep_alive_interval: None,
                http2_keep_alive_timeout: None,
                http2_keep_alive_while_idle: Parsed(
                    false,
                ),
                http1_only: Parsed(
                    true,
                ),
                http2_only: Parsed(
                    false,
                ),
            },
            sign_payload: true,
            skip_signature: true,
            disable_tagging: false,
            checksum: None,
            copy_if_not_exists: None,
            conditional_put: None,
            encryption_headers: S3EncryptionHeaders(
                {},
            ),
        },
        client: Client {
            accepts: Accepts,
            proxies: [
                Proxy(
                    System(
                        {},
                    ),
                    None,
                ),
            ],
            referer: true,
            default_headers: {
                "accept": "*/*",
                "user-agent": "object_store/0.11.0",
            },
            timeout: 30s,
        },
    },
}

[tustvold]

The error shows that it has retried 5 times, can you enable tokio tracing and report the log output. This will tell us what error it is encountering

[rkimoakbioinformatics]

The following is what I got before the error.

2024-10-17T11:38:43.648009Z  INFO ThreadId(01) /Users/user1/.cargo/registry/src/index.crates.io-6f17d22bba15001f/object_store-0.11.0/src/client/retry.rs:358: Encountered transport error backing off for 0.1 seconds, retry 1 of 10: error sending request for url (https://s3.us-east-1.amazonaws.com/testbucket/testfile.txt)
2024-10-17T11:39:13.750287Z  INFO ThreadId(01) /Users/user1/.cargo/registry/src/index.crates.io-6f17d22bba15001f/object_store-0.11.0/src/client/retry.rs:358: Encountered transport error backing off for 0.16576494 seconds, retry 2 of 10: error sending request for url (https://s3.us-east-1.amazonaws.com/testbucket/testfile.txt)
2024-10-17T11:39:43.919116Z  INFO ThreadId(01) /Users/user1/.cargo/registry/src/index.crates.io-6f17d22bba15001f/object_store-0.11.0/src/client/retry.rs:358: Encountered transport error backing off for 0.22046801 seconds, retry 3 of 10: error sending request for url (https://s3.us-east-1.amazonaws.com/testbucket/testfile.txt)
2024-10-17T11:40:14.143353Z  INFO ThreadId(01) /Users/user1/.cargo/registry/src/index.crates.io-6f17d22bba15001f/object_store-0.11.0/src/client/retry.rs:358: Encountered transport error backing off for 0.16017044 seconds, retry 4 of 10: error sending request for url (https://s3.us-east-1.amazonaws.com/testbucket/testfile.txt)
2024-10-17T11:40:44.306920Z  INFO ThreadId(01) /Users/user1/.cargo/registry/src/index.crates.io-6f17d22bba15001f/object_store-0.11.0/src/client/retry.rs:358: Encountered transport error backing off for 0.30901533 seconds, retry 5 of 10: error sending request for url (https://s3.us-east-1.amazonaws.com/testbucket/testfile.txt)

[tustvold]

Are you relying on some sort of proxy or similar? What happens if you try to communicate with that URL using something like curl?

(I presume you've redacted the URL as opposed to trying to communicate with a bucket called testbucket)

[rkimoakbioinformatics]

Using curl and the same URL, I can immediately see the content of the file.

One thing I do is that I use tokio::runtime::Runtime::block_on(async { ... } ) to run the code in my first post.

[tustvold]

This works fine on my computer

#[test]
fn test() {
    let runtime = tokio::runtime::Builder::new_current_thread()
        .enable_all()
        .build()
        .unwrap();

    runtime.block_on(async move {
        let store = crate::aws::AmazonS3Builder::new()
            .with_region("us-east-1")
            .with_bucket_name("testbucket")
            .with_skip_signature(true)
            .build()
            .unwrap();

        let mut stream = store.get(&Path::from("testfile.txt")).await;
        dbg!(&stream);
    })
}

And prints a 403 permission denied

[rkimoakbioinformatics]

I found that the problem was caused by passing a tokio runtime over FFI. I don't see any problem when I used a runtime which was created in the same function. Thanks for your help!

[rkimoakbioinformatics]

What I meant by FFI is Rust libraries dynamically loaded with libloading.