ENOENT: no such file or directory and EPERM: operation not permitted

[CoreyCole]

I'm on OSX 10.13.6 with Docker version 19.03.1. I've been using serverless for deploying my cloudformation script and now I'm attempting to run my stack locally using serverless-localstack.

I'm running the command

serverless deploy --verbose --stage local

error 1

Serverless: Using serverless-localstack
Serverless: Compiling with Typescript...
Serverless: Using local tsconfig.json
Serverless: Typescript compiled.
Serverless: Skip plugin function Package.packageService (lambda.mountCode flag is enabled)

  Error --------------------------------------------------

  ENOENT: no such file or directory, stat 'xxx/lambdas/.build/.serverless'

     For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.

  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Issues:        forum.serverless.com

  Your Environment Information ---------------------------
     Operating System:          darwin
     Node Version:              8.12.0
     Serverless Version:        1.49.0
     Enterprise Plugin Version: 1.3.5
     Platform SDK Version:      2.1.0

Right after this failure, I try to deploy again using the same command now that there is a .build directory that was created.

error 2 (after running deploy a 2nd time)

Serverless: Using serverless-localstack
Serverless: Compiling with Typescript...
Serverless: Using local tsconfig.json
Serverless: Typescript compiled.

  Error --------------------------------------------------

  EPERM: operation not permitted, unlink 'xxx/lambdas/.build/node_modules'

     For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.

  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Issues:        forum.serverless.com

  Your Environment Information ---------------------------
     Operating System:          darwin
     Node Version:              8.12.0
     Serverless Version:        1.49.0
     Enterprise Plugin Version: 1.3.5
     Platform SDK Version:      2.1.0

If I deploy a 3rd time, or any more, I keep getting error 2.

If I now delete the .build directory and try the deploy command again, I get back to error 1 again, then 2.

serverless.yml

# For full config options, check the docs: https://serverless.com/framework/docs/providers/aws/guide/serverless.yml/
#    docs.serverless.com

service: name

plugins:
- serverless-plugin-typescript
- serverless-iam-roles-per-function
- serverless-localstack

custom:
localstack:
  host: http:://localhost
  stages: # list of stages for which the plugin should be enabled
    - local
  autostart: true # optional - start LocalStack in Docker on Serverless deploy
  endpoints:
    # This section is optional - can be used for customizing the target endpoints
    S3: http://localhost:4572
    DynamoDB: http://localhost:4570
    CloudFormation: http://localhost:4581
    Lambda: http://localhost:4574
  lambda: # Enable this flag to improve performance
    # https://github.com/localstack/serverless-localstack#mounting-lambda-code-for-better-performance
    mountCode: True
  docker: # Enable this flag to run "docker ..." commands as sudo
    sudo: False
serverless-iam-roles-per-function:
  defaultInherit: true
stageEnv: ${opt:stage, self:provider.stage}
dynamoTableName: name-${opt:stage}-Rooms
dynamoTableArn: !Join [ ':', [ 'arn:aws:dynamodb', !Ref 'AWS::Region', !Ref 'AWS::AccountId', 'table/${self:custom.dynamoTableName}' ] ]
createServerLambda: !Join [ '-', [ !Ref 'AWS::StackName', 'CreateServer' ] ]
clusterName: !Join [ '-', [ !Ref 'AWS::StackName', 'ServerCluster' ] ]
serverServiceName: !Join ['-', [ !Ref 'AWS::StackName', 'NameServerService' ] ]
serverDockerImage: !Join [ '.', [ !Ref 'AWS::AccountId', 'dkr.ecr.us-west-2.amazonaws.com/name_server:${opt:stage}' ] ]
serverTaskDefinitionName: name-${opt:stage}-NameServerTaskDefinition
serverTaskDefinitionArn: !Join [ ':', [ 'arn:aws:ecs', !Ref 'AWS::Region', !Ref 'AWS::AccountId', 'task-definition/${self:custom.serverTaskDefinitionName}' ] ]
fargateSecurityGroupId: !Ref PublicAccessSG
vpcId: vpc-xxx
subnetOneId: subnet-xxx
igwId: igw-xxx
rtbId: !Ref PublicRouteTable

provider:
name: aws
stage: ${opt:stage, 'dev'} # https://serverless.com/framework/docs/providers/aws/guide/variables#recursively-reference-properties
runtime: nodejs10.x
region: us-west-2
memorySize: 256
cfLogs: true
logRetentionInDays: 14
deploymentBucket:
  name: name-deployments
iamRoleStatements: # inherited by all lambda functions
# https://serverless.com/framework/docs/providers/aws/guide/iam/
# https://medium.com/@glicht/serverless-framework-defining-per-function-iam-roles-c678fa09f46d
  - Effect: Allow
    Action:
      - logs:CreateLogGroup
      - logs:CreateLogStream
      - logs:PutLogEvents
    Resource: !Join [ ':', [ 'arn:aws:logs', !Ref 'AWS::Region', !Ref 'AWS::AccountId', 'log-group:/aws/lambda/*:*:*' ] ]
  - Effect: Allow
    Action:
      - s3:PutObject
    Resource: arn:aws:s3:::name-deployments
environment:
  stageEnv: ${self:custom.stageEnv}
  dynamoTableName: ${self:custom.dynamoTableName}
  createServerLambda: ${self:custom.createServerLambda}
  clusterName: ${self:custom.clusterName}
  serverTaskDefinitionArn: ${self:custom.serverTaskDefinitionArn}
  fargateSecurityGroupId: ${self:custom.fargateSecurityGroupId}
  subnetOneId: ${self:custom.subnetOneId}

functions:
AddParticipant:
  handler: handlers/AddParticipant.addParticipant
  events:
    - http:
        path: addParticipant
        method: post
  iamRoleStatements:
    - Effect: Allow
      Action:
        - dynamodb:GetItem
        - dynamodb:Query
      Resource: ${self:custom.dynamoTableArn}
CreateRoom:
  handler: handlers/CreateRoom.createRoom
  events:
    - http:
        path: createRoom
        method: post
  iamRoleStatements:
    - Effect: Allow
      Action:
        - lambda:InvokeFunction
      Resource: !Join [ ':', [ 'arn:aws:lambda', !Ref 'AWS::Region', !Ref 'AWS::AccountId', 'function', '${self:custom.createServerLambda}' ] ]
    - Effect: Allow
      Action:
        - dynamodb:PutItem
        - dynamodb:DeleteItem
        - dynamodb:GetItem
        - dynamodb:Query
        - dynamodb:UpdateItem
      Resource: ${self:custom.dynamoTableArn}
CreateServer:
  handler: handlers/CreateServer.createServer
  events:
    - http:
        path: createServer
        method: post
  iamRoleStatements:
    - Effect: Allow
      Action:
        - ecs:RunTask
      Resource: '*'
    - Effect: Allow
      Action:
        - iam:PassRole
      Resource: '*'
ServerStateChanged:
  handler: handlers/ServerStateChanged.serverStateChanged
  events:
    - cloudwatchEvent: # https://serverless.com/framework/docs/providers/aws/events/cloudwatch-event/
        description: "CloudWatch Event triggered when a server's state changes to Running"
        name: ServerRunning-${opt:stage}
        event:
          source:
            - aws.ecs
          detail-type:
            - 'ECS Task State Change'
          detail:
            clusterArn:
              - !GetAtt 'ECSCluster.Arn'
            lastStatus:
              - RUNNING
            desiredStatus:
              - RUNNING
    - cloudwatchEvent:
        description: "CloudWatch Event triggered when a server's state changes to Stopped"
        name: ServerStopped-${opt:stage}
        event:
          source:
            - aws.ecs
          detail-type:
            - 'ECS Task State Change'
          detail:
            clusterArn:
              - !GetAtt 'ECSCluster.Arn'
            lastStatus:
              - DEPROVISIONING
            desiredStatus:
              - STOPPED
  iamRoleStatements:
    - Effect: Allow
      Action:
        - dynamodb:PutItem
        - dynamodb:DeleteItem
        - dynamodb:Scan
      Resource: ${self:custom.dynamoTableArn}
    - Effect: Allow
      Action:
        - ec2:DescribeNetworkInterfaces
      Resource: '*'

resources:
Resources:
  # fargate task log group
  NameServerLogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: !Join [ '-', [ /ecs/, !Ref 'AWS::StackName', 'ZServerTaskDefinition' ] ]
  NameServerTaskDefinition:
    Type: AWS::ECS::TaskDefinition
    DependsOn: NameServerLogGroup # make sure the log group is created before it is used.
    Properties:
      # name of the task definition. Subsequent versions of the task definition are grouped together under this name.
      Family: !Join [ '-', [ !Ref 'AWS::StackName', 'NameServerTaskDefinition' ] ]
      # awsvpc is required for Fargate
      NetworkMode: awsvpc
      RequiresCompatibilities:
        - FARGATE
      # 256 (.25 vCPU) - Available memory values: 0.5GB, 1GB, 2GB
      # 512 (.5 vCPU) - Available memory values: 1GB, 2GB, 3GB, 4GB
      # 1024 (1 vCPU) - Available memory values: 2GB, 3GB, 4GB, 5GB, 6GB, 7GB, 8GB
      # 2048 (2 vCPU) - Available memory values: Between 4GB and 16GB in 1GB increments
      # 4096 (4 vCPU) - Available memory values: Between 8GB and 30GB in 1GB increments
      Cpu: 1024
      # 0.5GB, 1GB, 2GB - Available cpu values: 256 (.25 vCPU)
      # 1GB, 2GB, 3GB, 4GB - Available cpu values: 512 (.5 vCPU)
      # 2GB, 3GB, 4GB, 5GB, 6GB, 7GB, 8GB - Available cpu values: 1024 (1 vCPU)
      # Between 4GB and 16GB in 1GB increments - Available cpu values: 2048 (2 vCPU)
      # Between 8GB and 30GB in 1GB increments - Available cpu values: 4096 (4 vCPU)
      Memory: 2GB
      # The ARN of the task execution role that containers in this task can assume. 
      # All containers in this task are granted the permissions that are specified in this role.
      ExecutionRoleArn: !GetAtt 'FargateTaskExecutionRole.Arn'
      ContainerDefinitions:
        - Name: ${self:custom.serverServiceName}
          Image: ${self:custom.serverDockerImage}
          PortMappings:
            - ContainerPort: 3000
          # Send logs to CloudWatch Logs
          LogConfiguration:
            LogDriver: awslogs
            Options:
              awslogs-region: !Ref AWS::Region
              awslogs-group: !Ref NameServerLogGroup
              awslogs-stream-prefix: ecs
  # ECS cluster which spins up fargate servers
  ECSCluster:
    Type: AWS::ECS::Cluster
    Properties:
      ClusterName: ${self:custom.clusterName}
      Tags:
        - Key: name
          Value: name-serverless-ecs-cluster
  # role for the fargate task itself
  FargateTaskExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Join [ '-', [ !Ref 'AWS::StackName', 'FargateTaskExecutionRole' ] ]
      AssumeRolePolicyDocument:
        Statement:
          - Effect: Allow
            Principal:
              Service: ecs-tasks.amazonaws.com
            Action: 'sts:AssumeRole'
      ManagedPolicyArns:
        # allows fargate task to pull ECR image and write to cloudwatch logs
        - 'arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy'
  # security group used by lambda when spinning up a server fargate task (fully open)
  PublicAccessSG:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Access to and from the public
      VpcId: ${self:custom.vpcId}
      SecurityGroupIngress:
        # Allow inbound access from anywhere on the internet
        - CidrIp: 0.0.0.0/0
          IpProtocol: -1
      SecurityGroupEgress:
        # Allow outbound access from anywhere on the internet
        - CidrIp: 0.0.0.0/0
          IpProtocol: -1
      Tags:
        - Key: name
          Value: name-serverless-public-access-security-group

  # DynamoDB
  Rooms:
    Type: AWS::DynamoDB::Table
    Properties:
      TableName: ${self:custom.dynamoTableName}
      BillingMode: PAY_PER_REQUEST
      AttributeDefinitions:
        - AttributeName: roomId
          AttributeType: S
        - AttributeName: ipAddress
          AttributeType: S
      KeySchema:
        - AttributeName: roomId
          KeyType: HASH # partition key (can be multiple for given partition key, i.e. "available")
        - AttributeName: ipAddress
          KeyType: RANGE # sort key (unique & can be "available")

Outputs:
  ECSClusterArn:
    Description: The ARN of the ECS cluster
    Value: !GetAtt 'ECSCluster.Arn'
  FargateTaskDefinitionArn:
    Description: The ARN of the server task definition
    Value: ${self:custom.serverTaskDefinitionArn}
  RoomsDynamoDBTableArn:
    Description: The ARN of the DynamoDB table for managing room IP addresses
    Value: !GetAtt 'Rooms.Arn'
  DockerArn:
    Description: The ARN of the sever docker image
    Value: ${self:custom.serverDockerImage}

I've tried (unsuccssfully) rm -rf node_module/ && npm install --no-bin-links as suggested here

[CoreyCole]

I restarted my computer because I found it might be related to some process trying to also use the same files (likely some serverless localstack docker process I ctrl + C'd). So now I'm no longer getting the above errors, but now it permanently hangs at:

Serverless: Starting LocalStack in Docker. This can take a while.

(I've left it at this point for >30 minutes)

[CoreyCole]

To fix the hanging problem, I did

> docker ps -a

CONTAINER ID        IMAGE                   COMMAND                  CREATED             STATUS              PORTS               NAMES
0310a57387f0        localstack/localstack   "docker-entrypoint.sh"   30 minutes ago      Created                                 localstack_main

docker rm 031

And then

serverless deploy --verbose --stage local

And I'm back at the errors again

Serverless: Starting LocalStack in Docker. This can take a while.
Serverless: Checking state of LocalStack container dd2bf46379c2
Serverless: Checking state of LocalStack container dd2bf46379c2
Serverless: Checking state of LocalStack container dd2bf46379c2
Serverless: Checking state of LocalStack container dd2bf46379c2
Serverless: Checking state of LocalStack container dd2bf46379c2
Serverless: Checking state of LocalStack container dd2bf46379c2
Serverless: Checking state of LocalStack container dd2bf46379c2
Serverless: Checking state of LocalStack container dd2bf46379c2
Serverless: Checking state of LocalStack container dd2bf46379c2
Serverless: Using serverless-localstack
Serverless: Compiling with Typescript...
Serverless: Using local tsconfig.json
Serverless: Typescript compiled.

  Error --------------------------------------------------

  EPERM: operation not permitted, unlink '/Users/corey/Workspace/omnivor-io/omnimic/lambdas/.build/node_modules'

     For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.

  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Issues:        forum.serverless.com

  Your Environment Information ---------------------------
     Operating System:          darwin
     Node Version:              8.12.0
     Serverless Version:        1.49.0
     Enterprise Plugin Version: 1.3.5
     Platform SDK Version:      2.1.0

Serverless: Creating deployment bucket omnimic-deployments

[karvapallo]

This issue is caused by using the serverless-plugin-typescript with the serverless-localstack plugin. I'm getting the same error with the most basic one lambda setup using TypeScript. You might want to post this issue to https://github.com/prisma-labs/serverless-plugin-typescript/issues as well.

P.S. There seems to already be an issue posted there serverless/serverless-plugin-typescript#170

[CoreyCole]

I'm no longer trouble shooting this issue because we decided to not use local stack

[fsproru]

@whummer great plugin! We're deploying a test system into LocalStack with Commandeer and running into the same issue. It deploys just fine into a real AWS account.

Do you have any pointers or now to resolve this issue when using serverless-localstack?

[whummer]

Reopening this issue - we'll look into it shortly @fsproru . Can you please share a small, self-contained example to help us reproduce this issue? Thanks

[fsproru]

Thanks @whummer, you can reproduce the issue by deploying our Open Source Tanks project into LocalStack here: https://github.com/commandeer/open/pull/25/files

In order to reproduce it, you can go to serverless folder and run yarn install and sls deploy --stage local

Let me know if you have any questions.

[whummer]

Thanks for providing a sample project @fsproru . Many thanks also to @CoreyCole @karvapallo for providing the initial analysis.

This should be fixed in #56 - can you please give it a try with the latest version 0.4.21 of the npm package.

Note: Depending on your setup, you may have to downgrade serverless-plugin-typescript to version 1.1.7 in order to fix this issue: serverless/serverless-plugin-typescript#170 . It may also work with the latest version of that plugin. Please let us know, thanks

[fsproru]

@whummer this issue is fixed, thanks a bunch!

You're right, I had to set serverless-plugin-typescript to 1.1.7 and serverless-localstack to 0.4.21.

We still have an issue deploying the Tanks open-source project which deploys just fine to an AWS account. I'll submit a different issue so we can close this one.

[whummer]

Thanks for confirming this works. Please report here if the problem persists. Going to tackle the other issue in a bit.