Handling all possible errors from AWS / Boto

[Hvass-Labs]

Describe the issue

Hello,

I am using AWS SES to send e-mails. I am new to AWS and it is unclear from the docs how to handle all possible errors arising from AWS / Boto, so they can be logged and investigated later. The user doesn't need to know exactly what happened and will just see a generic error-message.

My code is essentially:

try:
    client = boto3.client('sesv2', region_name=AWS_REGION_NAME,
                          aws_access_key_id=AWS_ACCESS_KEY_ID,
                          aws_secret_access_key=AWS_SECRET_ACCESS_KEY)
                          
    try:
        response = client.send_email(...)
    finally:
        client.close()

    # Do I also have to check response['ResponseMetadata']['HTTPStatusCode'] != 200 ?

except botocore.exceptions.ClientError as e:
    logging.error(str(e))

except botocore.exceptions.BotoCoreError as e:
    logging.error(str(e))

Are these two exception classes sufficient to handle all possible errors arising in AWS / Boto?

Or do I also need to inspect the response object for other possible errors if the HTTP status-code isn't 200 for success?

Thanks!

Links

https://boto3.amazonaws.com/v1/documentation/api/latest/guide/error-handling.html

[tim-finnigan]

Hi @Hvass-Labs thanks for reaching out. As mentioned in the error handling documentation that you referenced, it notes:

The most common botocore exception you’ll encounter is ClientError. This is a general exception when an error response is provided by an AWS service to your Boto3 client’s request.

So while it is possible to account for each specific exception, ClientError is the main one to account for. You can see all of the static exceptions in this toggle list on that documentation page:

And the full list of possible exceptions can be found in the exceptions.py file. For example BotoCoreError, which serves as a catchall for unspecified Botocore errors, but wouldn't account for things like networking-related errors.

I hope that helps - if you have any more questions please let us know. Or if you are encountering issues other than ClientError, we could look into those further. If you add boto3.set_stream_logger('') to your script, then that will print a full log stack trace to help give more insight into any issues.

[Hvass-Labs]

Thanks for the detailed reply! I did actually read the doc-page and noticed the section you highlight :-)

But it is still a bit unclear to me, so allow me to ask more specifically:

1. Is it necessary to catch both BotoCoreError and ClientError to make sure I catch all custom exceptions that can be raised by Boto / AWS?

2. The client.send_mail() function returns a response object. Is it necessary to also check it for HTTP error-codes e.g. response['ResponseMetadata']['HTTPStatusCode'] != 200, or will all HTTP error-codes cause the Boto library to raise an exception?

Thanks for the tip on logging the stack-trace. However, as this is running on a web-server, I don't log the stack-trace because I am worried it might contain the AWS account credentials, which I don't want in the log for security reasons. But perhaps I'm wrong?

[tim-finnigan]

Thanks for following up and your patience. I think the snippet below helps show what you can do here.

You can use ClientError for general exceptions when an error response is provided by an AWS service to your Boto3 client’s request. If you want to catch specific exceptions like ParamValidationError or others from that documented list, then you can also specify those. BotoCoreError is a catchall for unspecified Botocore errors, so won't catch everything, but you can just use except Exception as error: to catch any errors.

import boto3
import botocore

ses_client = boto3.client("sesv2")

# Define the email parameters
# ...

try:
    response = ses_client.send_email(
        FromEmailAddress=sender_email,
        Destination=destination,
        Content=message
    )
    print("Email sent! Message ID:", response['MessageId'])

except botocore.exceptions.ClientError as error:
    # Handle specific AWS service errors
    if error.response['Error']['Code'] == 'MessageRejected':
        print("Error: Message rejected by Amazon SES:", error.response['Error']['Message'])
    else:
        print("An error occurred while sending the email:", error)

except botocore.exceptions.ParamValidationError as error:
    raise ValueError('The parameters you provided are incorrect: {}'.format(error))

except Exception as error:
    # Handle any other exceptions
    print("An unexpected error occurred:", error)

---

Also in case you didn't see it I will highlight this section on parsing error responses, with I think this part being the most relevant here:

When attempting to catch AWS service exceptions, one way is to catch ClientError and then parse the error response for the AWS service-specific exception.

So as long as you're catching ClientError then it catches the modeled service API exception, and you can parse the error response if you want to see the specific exception.

---

For more information on logging please see the boto3.set_stream_logger documentation. As mentioned there:

For debugging purposes a good choice is to set the stream logger to '' which is equivalent to saying “log everything”.

But also note the Warning:

Be aware that when logging anything from 'botocore' the full wire trace will appear in your logs. If your payloads contain sensitive data this should not be used in production.

[github-actions[bot]]

Greetings! It looks like this issue hasn’t been active in longer than five days. We encourage you to check if this is still an issue in the latest release. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or upvote with a reaction on the initial post to prevent automatic closure. If the issue is already closed, please feel free to open a new one.

[Hvass-Labs]

Thanks for the example. Maybe I am being a stickler, but I'm still not sure it answers my two questions above :-) Would it be possible to give yes / no answers to those two questions?

My project is running on a web-server where I give broad error-messages to the user for different kinds of error-types. I am catching both BotoCoreError and ClientError, as well as many other error-types. As long as there aren't any more errors reported in the Boto response object as in my 2nd question above, I should be fine.

[tim-finnigan]

Sure I'll try to clarify:

1. Is it necessary to catch both BotoCoreError and ClientError to make sure I catch all custom exceptions that can be raised by Boto / AWS?

No - to catch every exception, you would need to catch every documented exception. As mentioned in the docs, ClientError is the most common.

2. The client.send_mail() function returns a response object. Is it necessary to also check it for HTTP error-codes e.g. response['ResponseMetadata']['HTTPStatusCode'] != 200, or will all HTTP error-codes cause the Boto library to raise an exception?

No - it isn't necessary to also check for HTTP error codes. But depending on the service/API and your use case, you may want to check. For example with SNS the PublishBatch API documentation says:

Because the batch request can result in a combination of successful and unsuccessful actions, you should check for batch errors even when the call returns an HTTP status code of 200.

In the case of S3 copy requests, if an error is found with a 200 status code then it is changed here to 500.

Converting this to a Q&A discussion. If you want to create a new issue with a specific request please feel free to.

[Hvass-Labs]

Thank you! I guess it's a complicated system :-)