S3AsyncClient issue

[AceHack]

When trying to perform any action using the S3AsyncClient I get the following error. Works fine with real S3 server also, works fine when using S3Client and s3mock.

[WARN] [06/07/2019 17:02:57.707] [s3mock-akka.actor.default-dispatcher-4] [akka.actor.ActorSystemImpl(s3mock)] Illegal request, responding with status '400 Bad Request': Request is missing required Host header

• should ingest into solr from s3 based paths *** FAILED *** (1 second, 398 milliseconds)
  java.util.concurrent.CompletionException: software.amazon.awssdk.services.s3.model.S3Exception: null (Service: S3, Status Code: 400, Request ID: null)
  at software.amazon.awssdk.utils.CompletableFutureUtils.errorAsCompletionException(CompletableFutureUtils.java:61)
  at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncExecutionFailureExceptionReportingStage.lambda$execute$0(AsyncExecutionFailureExceptionReportingStage.java:51)
  at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:822)
  at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:797)
  at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
  at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
  at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:75)
  at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
  at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
  at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
  ...
  Cause: software.amazon.awssdk.services.s3.model.S3Exception: null (Service: S3, Status Code: 400, Request ID: null)
  at software.amazon.awssdk.services.s3.model.S3Exception$BuilderImpl.build(S3Exception.java:95)
  at software.amazon.awssdk.services.s3.model.S3Exception$BuilderImpl.build(S3Exception.java:55)
  at software.amazon.awssdk.protocols.query.unmarshall.AwsXmlErrorProtocolUnmarshaller.handle(AwsXmlErrorProtocolUnmarshaller.java:127)
  at software.amazon.awssdk.protocols.query.unmarshall.AwsXmlErrorProtocolUnmarshaller.handle(AwsXmlErrorProtocolUnmarshaller.java:86)
  at software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler.lambda$prepare$0(AsyncResponseHandler.java:88)
  at java.util.concurrent.CompletableFuture.uniCompose(CompletableFuture.java:952)
  at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:926)
  at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
  at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
  at software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler$BaosSubscriber.onComplete(AsyncResponseHandler.java:129)

[AceHack]

I should mention I'm using the AnonymousCredentialsProvider.

[sworisbreathing]

@AceHack I get this issue as well.
This was logged previously as aws/aws-sdk-java-v2#429 however I'm seeing it even with version 2.9.15 of the aws sdk.

Stepping through the code, it appears AWS's Apache httpclient implementation implicitly sets the host header, but the netty implementation doesn't and neither does the native java implementation

[sworisbreathing]

Update: I'm still digging into the AWS SDK code to find out where it's forgetting to set the host header. From the tests I'm running it seems to affect the S3AsyncClient but not DynamoDbAsyncClient or SqsAsyncClient, so I'm not entirely sure the problem is in the Netty Implementation (which is linked above this comment).

In the meantime, here's a workaround you can do in your tests:

Map<String, List<String>> S3_HOST_HEADERS = new HashMap<>();
S3_HOST_HEADERS.put("Host", Collections.singletonList("localhost:" + S3_PORT));

S3AsyncClient client = S3AsyncClient.builder()
  // ... other configs here
  .overrideConfiguration(builder -> builder.headers(S3_HOST_HEADERS))
  .serviceConfiguration(builder -> builder.pathStyleAccessEnabled(Boolean.TRUE))
  .build();

[sworisbreathing]

Update: found the issue, reported as aws/aws-sdk-java-v2#1473

Alternative workaround to the above is to use something other than anonymous credentials when setting up the S3AsyncClient. This will ensure that the underlying HTTP requests are properly signed, which will implicitly set the Host header. S3Mock doesn't seem to care what you provide, credentials-wise, so ideally you would just use:

S3AsyncClient client = S3AsyncClient.builder()
    .credentialsProvider(StaticCredentialsProvider.create(AwsBasicCredentials.create("x", "x")))
    .build();

However due to #10 this doesn't seem to work very well. So in the end I'd still stick with anonymous authentication and explicitly overriding the host header