sam local start-api fails to move /var/rapid/aws-lambda-rie-x86_64 in docker image for serverless function

[llacroix]

Description:

Accessing a lambda using a custom docker image with python awslambdaric fails with:

samcli.commands.local.cli_common.user_exceptions.ImageBuildException: Error building docker image: The command '/bin/sh -c mv /var/rapid/aws-lambda-rie-x86_64 /var/rapid/aws-lambda-rie && chmod +x /var/rapid/aws-lambda-rie' returned a non-zero code: 1

Steps to reproduce:

Create a serverless function with a custom docker image that uses awslambdaric try to access it through an HttpApi locally.

Observed result:

Invoking Container created from httphandler:latest
Image was not found.
Removing rapid images for repo httphandler
Building image..............
Failed to build Docker Image
NoneType: None
Exception on / [GET]
Traceback (most recent call last):
  File "flask/app.py", line 2447, in wsgi_app
  File "flask/app.py", line 1952, in full_dispatch_request
  File "flask/app.py", line 1821, in handle_user_exception
  File "flask/_compat.py", line 39, in reraise
  File "flask/app.py", line 1950, in full_dispatch_request
  File "flask/app.py", line 1936, in dispatch_request
  File "samcli/local/apigw/local_apigw_service.py", line 357, in _request_handler
  File "samcli/commands/local/lib/local_lambda.py", line 144, in invoke
  File "samcli/lib/telemetry/metric.py", line 230, in wrapped_func
  File "samcli/local/lambdafn/runtime.py", line 177, in invoke
  File "samcli/local/lambdafn/runtime.py", line 88, in create
  File "samcli/local/docker/lambda_container.py", line 94, in __init__
  File "samcli/local/docker/lambda_container.py", line 236, in _get_image
  File "samcli/local/docker/lambda_image.py", line 164, in build
  File "samcli/local/docker/lambda_image.py", line 278, in _build_image
samcli.commands.local.cli_common.user_exceptions.ImageBuildException: Error building docker image: The command '/bin/sh -c mv /var/rapid/aws-lambda-rie-x86_64 /var/rapid/aws-lambda-rie && chmod +x /var/rapid/aws-lambda-rie' returned a non-zero code: 1
2022-05-19 01:01:55 127.0.0.1 - - [19/May/2022 01:01:55] "GET / HTTP/1.1" 502 -

Expected result:

Should just work or tell how to fix the issue with moving rapid/aws-lambda-rie

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: Fedora
2. sam --version: 1.50.0
3. AWS region: us-east-1

[torresxb1]

Thanks for bringing this to our attention! Could you include a basic sample template, handler, and Dockerfile (and anything else necessary) so that we could reproduce this issue more easily and accurately?

[llacroix]

Yes, I'll give it a look when I'll have time

[mjvirt]

I have just run into this issue and it affects sam local invoke etc.

TL;DR: Lambda functions default to Linux/x86_64 architecture even when run locally with SAM. On Linux for arm/aarch(64) only the Docker Engine/CE is available to install. Unlike for Docker Desktop, CE doesn't include QEMU emulation configuration for other platforms (specifically Linux/x86_64). Third-party binfmt is needed. This on very light testing appears to resolve the issue for SAM.

Longer version:

docker ps -a and info shows the command that fails:

"mv /var/rapid/aws-lambda-rie-x86_64 /var/rapid/aws-lambda-rie && chmod +x /var/rapid/aws-lambda-rie"

Additional environment details:

OS: Fedora Linux 36 (ARM64) - Parallels VM on Macbook M1
sam --version: SAM CLI, version 1.56.0
AWS region: eu-central-1

It is fairly easy to reproduce. Just create the hello-world sam app via $ sam init in an empty directory:
1 - AWS Quick Start Templates
1 - Hello World Example

$ sam local invoke HelloWorldFunction
Invoking app.lambda_handler (python3.9)
Image was not found.
Removing rapid images for repo public.ecr.aws/sam/emulation-python3.9
Building image...................
Failed to build Docker Image
NoneType: None
Error: Error building docker image: The command '/bin/sh -c mv /var/rapid/aws-lambda-rie-x86_64 /var/rapid/aws-lambda-rie && chmod +x /var/rapid/aws-lambda-rie' returned a non-zero code: 1

The problem is due to a platform mismatch - attempting to run x86_64 images on arm64 architecture. Ordinarily with Docker Desktop on x86_64 it's possible to emulate other architectures such as arm64.

Per the AWS SAM document here https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-install-linux.html it states:

On Linux, to build and run Lambda functions with a different instruction set architecture than your host machine, you must take additional steps to configure Docker. For example, to run arm64 functions on an x86_64 machine, you can run the following command to configure the Docker daemon: docker run --rm --privileged multiarch/qemu-user-static --reset -p yes.

However, multiarch/qemu-user-static only supports the x86_64 host architecture per https://github.com/multiarch/qemu-user-static/.

I did find this and have managed to get it working for SAM: https://github.com/tonistiigi/binfmt.

[wangzhe3224]

Hi I am running into same issue, any update on the thread please?

[pub-technology]

Same issue, any solution for this problem

Note: I can build and deploy to the AWS without errors. However at local the error happen when trigger sam local invoke command line.

 Error building docker image: The command '/bin/sh -c mv /var/rapid/aws-lambda-rie-x86_64 
/var/rapid/aws-lambda-rie && chmod +x /var/rapid/aws-lambda-rie' returned a non-zero code: 1

[pub-technology]

What is the workaround solution you mentioned in this issue? @jensenmo
I'd be happy to help find a way to get your issue resolved as quickly as possible if you need me provide more info let me know

[mjvirt]

@wangzhe3224 @pub-technology Does the fix I described above not work for you?

#3891 (comment)

[chrisrhymes]

Hi @mjvirt I'm also having this same issue when using an M1 Mac running macOS 13.1.

I changed the template.yml Architecture to arm64, reran sam build and then sam local invoke --no-event worked for me.

Resources:
  HelloWorldFunction:
    Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
    Properties:
      CodeUri: hello-world/
      Handler: app.lambdaHandler
      Runtime: nodejs18.x
      Architectures:
        - arm64 # Update this line for it to work locally

[sbilello]

It would be great it if works for golang too

[mjvirt]

Hi @chrisrhymes and others, maybe I wasn't clear in my original message above, but this last line works:

I did find this and have managed to get it working for SAM: https://github.com/tonistiigi/binfmt (other CPU architecture emulation on arm64/M1)

In other words you can emulate x86_64 on arm64 - and it works fine with SAM/CDK - at least insofar as I've tested it (just use docker run --privileged --rm tonistiigi/binfmt --install all to install the available platform emulators; or you can be more picky...).

[chrisrhymes]

Thanks, I should have said before, but I tried running the command, but only for arm64 --install arm64 and it still didn't work.

I just tried the docker run --privileged --rm tonistiigi/binfmt --install all command which all looks ok, ran sam build and then sam local start-api and I get the same error as before:

Error building docker image: The command '/bin/sh -c mv /var/rapid/aws-lambda-rie-x86_64 /var/rapid/aws-lambda-rie && chmod +x /var/rapid/aws-lambda-rie' returned a non-zero code: 1

[ixolt]

@chrisrhymes I managed to fix this by executing the following commands

$ brew uninstall aws-sam-cli
$ brew install rust
$ brew install aws/tap/aws-sam-cli

[pub-technology]

@chrisrhymes I managed to fix this by executing the following commands

$ brew uninstall aws-sam-cli
$ brew install rust
$ brew install aws/tap/aws-sam-cli

Got an error from the last command brew install aws/tap/aws-sam-cli

Already downloaded: /Users/haithai/Library/Caches/Homebrew/downloads/563186ab187d77b0c27b2db3c0f3fc5d25300867da5d18083dd099e13ce4431b--aws-aws-sam-cli-v1.71.0-0-gc681067.tar.gz
Error: aws-sam-cli: SHA256 mismatch
Expected: c27333fb067fd0ffef723c48ed9f26272061d6f3299f5b00ee3d421cc607e0c5
  Actual: e05774257d3bbf0080d7d0f0114991bfaf44a9d5bc464cc1427ccc98a4c99cd6
    File: /Users/haithai/Library/Caches/Homebrew/downloads/563186ab187d77b0c27b2db3c0f3fc5d25300867da5d18083dd099e13ce4431b--aws-aws-sam-cli-v1.71.0-0-gc681067.tar.gz
To retry an incomplete download, remove the file above

I tried to remove and run install again - The error still happening @ixolt

[pub-technology]

Thanks @ixolt

The new version of AWS SAM CLI - (ﾉ◕ヮ◕)ﾉ*:･ﾟ✧ The problem Fixed

$ brew uninstall aws-sam-cli
$ brew install rust
$ brew tap aws/tap
$ brew install aws-sam-cli

[garyaparker]

I'm having the same issue on SAM 1.71.0 (latest) and the emulator install is not resolving it.

[sbilello]

I solved it by

$ rm -rf /usr/local/bin/sam

Installing sam via pkg file from https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/install-sam-cli.html
MacBook Pro M1

[cerdman]

Install from the package was the only thing that worked for me to be able to invoke locally with Jetbrains toolkit (M1 on Monterey). I did run the emulator install noted above prior to package install (unsure if this is a factor or not).

[sbilello]

The remaining problem is running golang 1.x in M1 mac so far and debug with GoLang and the AWS Toolkit

[luisdelarosa]

For an M1 Mac, you can pass the architecture flag -a to sam init like: sam init -a arm64.

Or you could change your Architectures line in your template.yaml like @chrisrhymes described above.

I wrote about it a bit more in this Stack Overflow answer: https://stackoverflow.com/questions/75216548/aws-sam-cli-throws-error-error-building-docker-image/75342572#75342572

I think the ideal solution would be for SAM to detect your architecture and act accordingly when creating a Docker image for emulating Lambda locally, while allowing for the Architectures to specify the deployment architecture of the Lambda function. Currently it looks SAM uses the Architectures line for both.

[sriram-mv]

Moving this issue to discussions: https://github.com/aws/aws-sam-cli/discussions/4705

Couple of things to consider, specifying the architecture in template.yaml for making it work for local emulation also means that its the same architecture that will be used for deployment. Ideally its best to use arm64 when working on M1/M2 series of macs to ensure a smooth local workflow.

However, I do like the suggestion on sam inits being smarter to set the appropriate architecture organically.

Converting this to a discussion as its broad ranging in scope with respect to architecture of the underlying host.

[sbilello]

Did we have any progress on this?

[AGSaidi]

Aren't we just talking about taking this
defaulting the build to X86_64 if nothing is specified and changing that to build on Arm64 if nothing in specified and we're on an Arm64 host?

[AGSaidi]

#4730 does this

[jfuss]

I don't think this is the behavior we want in SAM CLI though. If we default to the host system, you are testing on arm but risk the deployment or things not working as expected once deployed (since if Arch is not defined AWS Lambda defaults to x86). As of now, SAM CLI operates based on the Template which is to ensure you are testing/build for the target Arch on AWS Lambda (not your system).

[sbilello]

@jfuss I can understand your point of view but I guess that all the developers working on their Mac with M1/M2 chip and willing to debug Lambda with their IDEs (VSCode or GoLand) needs to be supported in some way.
Maybe we can override the cloudformation config with --local parameter that would consider the host architecture instead of the architecture defined in cloudformation? We can even find a better name for it

[jfuss]

@sbilello So cross os is supported. Check out my comment below: #4708 (comment). I have a guess at what might be going on here and why we have had a hard time tracking down what is really going on.

[sbilello]

I didn't use brew to install the sam cli @jfuss

[pstephenwille]

Same here on an M1.

@sriram-mv comment matches my errors.

Changing to arm64 and running Java8 which looks to be too old for Arm64.

[jfuss]

Out of curiosity, how are people installing SAM CLI that are running into this?

If it is by brew, can you uninstall aws-sam-cli and then do brew tap aws/tap & brew install aws/tap/aws-sam-cli or using our dmg installer (the preferred method assuming MacOS) following https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/install-sam-cli.html.

I reached out to Homebrew but they added AWS SAM CLI to the core but it's incorrect and removes a file they shouldn't be (the rie version that does not match the system). Which I have a hutch is why this is occurring for folks more and more often, and why we have had a hard time narrowing things down.

[AGSaidi]

In my case pip but I'm not using mac os to do so.