Async clients do not send host header when using anonymous credentials

[sworisbreathing]

Expected Behavior

All HTTP requests sent using HTTP 1.1 send a valid HTTP host header, unless explicitly overridden in the client configuration.

Current Behavior

When using the default SdkAsyncClient implementations with anonymous credentials, requests are sent using HTTP 1.1 but do not include the mandatory Host header.

Stepping through the code in a debugger session, it appears that the host header is being set by AbstractAws4Signer, but that logic is skipped in BaseAws4Signer.sign() when anonymous credentials are used.

Possible Solution

• Add a step in the request pipeline in AmazonAsyncHttpClient.execute() to set a default host header if none was specified, and/or
• Recommendations in HTTP Host header not set with Netty/Native Java implementations #1467 (as a fallback)

Steps to Reproduce (for bugs)

1. Create an async client with anonynmous credentials, similar to the following:

    S3AsyncClient s3Client = S3AsyncClient.builder()
             .credentialsProvder(AnonymousCredentialsProvider.create())
             .build();

1. Send any request using the client, capturing the raw HTTP wire traffic.
2. Observe that the request claims to use HTTP 1.1 but does not send the mandatory Host header

Context

I'm trying to create some automated offline tests of an application which makes use of a number of AWS services such as DynamoDB, S3, and SQS. The tests use libraries such as S3Mock and DynamoDBLocal to mimic the AWS APIs without needing to hit an actual AWS account.

The tests are breaking because S3Mock does not require authentication, so I used anonymous credentials when setting up the S3 client, but S3Mock uses akka-http under the hood which strictly enforces the host header for HTTP 1.1 requests.

this issue is reported in findify/s3mock#147 as well.

Your Environment

• AWS Java SDK version used: 2.9.15
• JDK version used: 1.8 (doesn't matter)
• Operating System and version: Windows 10 (doesn't matter)

[dagnir]

Hi @sworisbreathing, this was discussed a little here: #1407 (comment). While this is valid behavior according to the spec, I think we will go ahead and make this change as it's much less surprising for users of the library for the Host header to be present.

[sworisbreathing]

Hi @dagnir, it seems I've read the spec a bit differently than you have:

• Section 5.1.2 only says that if it's not an absolute URI, then "the network location of the URI (authority) MUST be transmitted in a Host header field". It doesn't really talk about what the Host header should be when an absolute URI is used.
• Section 19.6.1.1 explicitly calls out that:

• A client that sends an HTTP/1.1 request MUST send a Host header.
• Servers MUST report a 400 (Bad Request) error if an HTTP/1.1 request does not include a Host request-header.

• Section 14.23 states:

A client MUST include a Host header field in all HTTP/1.1 request messages . If the requested URI does not include an Internet host name for the service being requested, then the Host header field MUST be given with an empty value.

I'll admit I find the last sentence a bit confusing and contradictory to the rest of the spec, but it seems that the header needs to be sent all the time, but there are certain times when it can be empty.

[sworisbreathing]

I've already opened #1478 which addresses this issue (at least partially)

[dagnir]

Thanks for the PR! I will have a look today.

Hmm yes the points under 19.6.1.1 seem a little contradictory. Here is the part where the spec mentions the Host header in the presence of an absoluteURI in 5.2:

1. If Request-URI is an absoluteURI, the host is part of the
   Request-URI. Any Host header field value in the request MUST be
   ignored.

[dagnir]

Fixed by #1500 as well.