curlCode: 60, SSL peer certificate or SSH remote key was not OK

[weiji14]

Version

1.44.0

Describe the bug.

Trying to read JPEG files from an s3 bucket, but getting some authentication related errors. Accessing the files using aws s3 ls works ok though (after I set some AWS_* environment variables). Is there some way I can pass through credentials/settings needed to access images in a non-public-facing bucket?

Minimum reproducible example

from nvidia.dali import fn
from nvidia.dali.pipeline import pipeline_def

@pipeline_def
def simple_pipeline(files):
    jpegs, _labels = fn.readers.file(files=files, name="Reader")
    return jpegs

files = ["s3://<bucket-name>/<folder-name>/<image_name>.jpeg"]
pipe = simple_pipeline(files=files, batch_size=2, num_threads=1, device_id=0)
pipe.build()

pipe_out = pipe.run()
print(pipe_out)

Relevant log output

Traceback (most recent call last):
  File "/home/ubuntu/project/temp.py", line 17, in <module>
    pipe_out = pipe.run()
               ^^^^^^^^^^
  File "/home/ubuntu/project/.venv/lib/python3.12/site-packages/nvidia/dali/pipeline.py", line 1396, in run
    return self.outputs(cuda_stream)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/project/.venv/lib/python3.12/site-packages/nvidia/dali/pipeline.py", line 1205, in outputs
    return self._outputs(cuda_stream)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/project/.venv/lib/python3.12/site-packages/nvidia/dali/pipeline.py", line 1314, in _outputs
    return self._pipe.Outputs(types._raw_cuda_stream_ptr(cuda_stream))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Critical error in pipeline:
Error in CPU operator `nvidia.dali.fn.readers.file`,
which was used in the pipeline definition with the following traceback:

  File "/home/ubuntu/project/temp.py", line 10, in simple_pipeline
    jpegs, _labels = fn.readers.file(files=files, name="Reader")

encountered:

S3 Object not found. bucket=<bucket-name> object=<folder-name>/<image-name>.jpeg:
: curlCode: 60, SSL peer certificate or SSH remote key was not OK
Current pipeline object is no longer valid.

Other/Misc.

Have tried looking at some Stackoverflow answers (e.g. at https://serverfault.com/questions/469824/curl-disable-certificate-verification), but they don't seem to work. A little unsure if it's something I need to resolve with curl or something in DALI.

Check for duplicates

• I have searched the open bugs/issues and have found no duplicates for this bug report

[JanuszL]

Hi @weiji14,

Thank you for reaching out. DALI uses AWS SDK, please check this short guide on how to set creds to it. It shouldn't be any different from your current approach with aws s3.

[jantonguirao]

We rely on the default credentials provider chain by AWS SDK, as Janusz mentioned.
Other than the AWS SDK environment variables, we expose AWS_ENDPOINT_URL to control the endpoint URL.
So, for example, you could set your credentials via environment variable:

export AWS_ENDPOINT_URL="your-endpoint-url"
export AWS_ACCESS_KEY_ID="your-access-key-id"
export AWS_SECRET_ACCESS_KEY="your-secret-access-key"
export AWS_REGION="your-region"

[weiji14]

Hmm, what version of AWS SDK CPP is DALI 1.44.0 using? I'm thinking if this might be a case of aws/aws-sdk-cpp#2908, because I've set AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_REGION and got aws s3 ls to work, but the DALI pipeline script ran in the same terminal is still returning curlCode: 60, SSL peer certificate or SSH remote key was not OK. It might well be some certificate issue that libcurl is more sensitive to.

[jantonguirao]

It uses aws-sdk-cpp 1.11.418 version.
One idea would be to recompile DALI enabling debug logs for AWS SDK to see if you get a more detailed message.