Skip to content

Commit

Permalink
Merge pull request #2475 from pjx98/pehjx-feature-apigw-lambda-s3-ter…
Browse files Browse the repository at this point in the history
…raform-java

New pattern: Upload files to S3 bucket using presigned URLs (terraform)
  • Loading branch information
julianwood authored Nov 15, 2024
2 parents d32e7a1 + a78c510 commit cba5beb
Show file tree
Hide file tree
Showing 10 changed files with 690 additions and 0 deletions.
107 changes: 107 additions & 0 deletions apigw-lambda-s3-terraform-java/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,107 @@
# Upload file to Amazon S3 bucket using presigned URL using Terraform

This sample project deploys an Amazon API Gateway REST API with an AWS Lambda integration. The Lambda function, written in Java, generates a presigned URL with write access to an Amazon S3 Bucket. The provided test class uploads a file up to 5GB into the S3 bucket using the presigned URL.

Learn more about this pattern at Serverless Land Patterns: serverlessland.com/patterns/apigw-lambda-s3-terraform-java

Alternatively, to implement this project using AWS SAM, refer to https://serverlessland.com/patterns/apigw-lambda-s3-sam-java

Important: this application uses various AWS services and there are costs associated with these services after the Free Tier usage - please see the [AWS Pricing page](https://aws.amazon.com/pricing/) for details. You are responsible for any AWS costs incurred. No warranty is implied in this example.

## Requirements

- [Create an AWS account](https://portal.aws.amazon.com/gp/aws/developer/registration/index.html) if you do not already have one and log in. The IAM user that you use must have sufficient permissions to make necessary AWS service calls and manage AWS resources.
- [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html) installed and configured
- [Git Installed](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git)
- [Terraform](https://learn.hashicorp.cxom/tutorials/terraform/install-cli?in=terraform/aws-get-started) installed
- [Java 21 or above](https://docs.aws.amazon.com/corretto/latest/corretto-21-ug/amazon-linux-install.html) installed
- [Maven 3.8.6 or above](https://maven.apache.org/download.cgi) installed


## Deployment Instructions

1. Create a new directory, navigate to that directory in a terminal and clone the GitHub repository:
```bash
git clone https://github.com/aws-samples/serverless-patterns
```

2. Change directory to the pattern directory:
```bash
cd serverless-patterns/apigw-lambda-s3-terraform-java
```

3. From the command line, execute the below command to build the Java based AWS Lambda funtion using Maven. This will create `MyS3PreSignedURLGeneratorFunction.zip` file under the `target` folder.
```bash
mvn clean package
```

4. From the command line, use Terraform to deploy the AWS resources for the pattern as specified in the main.tf file:
```bash
terraform init
terraform apply
```
5. During the prompts:

- Enter the desired prefix to identify your project
- Enter the desired AWS Region.
- Do you want to perform these actions? Enter 'yes'


6. Note the output from the terraform deployment process. This contains the REST API URL which is used for next step as well as testing.

## How it works

This Terraform project uses allows user to upload a file up to 5 GB size using Amazon S3 bucket presigned URL. Please refer to the architecture diagram below:

![End to End Architecture](images/architecture.png)

Here's a breakdown of the steps:

1. **Amazon API Gateway**: Receives the HTTP POST request containing the file name to be uploaded.

2. **AWS Lambda**: Triggered by the API Gateway, this function creates a presigned S3 bucket URL with PUT object permission for the provided key (file name).

3. **Amazon S3**: User uploads a file up to 5 GB size using presigned URL.

## Testing

1. For testing this, you can use any file up to 5 GB in size. For example you can download the [Amazon S3 user guide](https://docs.aws.amazon.com/pdfs/AmazonS3/latest/userguide/s3-userguide.pdf).


2. Run the following command from to run the test class to upload a file using Amazon S3 presigned URL:

```bash
java -cp target/test-classes/ com.example.S3UploadIntegrationTest
```

2. During the prompts:

- Enter the REST API URL to get the presigned URL: from the `terraform apply` output
- Enter the file name on S3 (key): your-file.pdf
- Enter the path to the local file: your-file.pdf

If everything goes fine you should get a confirmation message:
```bash
File uploaded successfully!
```

3. Log into [Amazon S3 Console]() and open the {prefix}-s3-upload bucket to validate the file got uploaded successfully.

## Cleanup

1. Delete the content in the Amazon S3 bucket using the following command. Please *ensure* that the correct bucket name is provided to avoid accidental data loss:
```bash
aws s3 rm s3://{prefix}-s3-upload --recursive --region {your-region}

2. To delete the resources deployed to your AWS account via Terraform, run the following command:

```bash
terraform destroy
```


---

Copyright 2024 Amazon.com, Inc. or its affiliates. All Rights Reserved.

SPDX-License-Identifier: MIT-0
91 changes: 91 additions & 0 deletions apigw-lambda-s3-terraform-java/apigw-lambda-s3-terraform-java.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
{
"title": "Upload file to Amazon S3 bucket using presigned URL",
"description": "Amazon API Gateway with AWS Lambda integration to showcase how to upload a file up to 5GB to Amazon S3 bucket using a presigned S3 URL.",
"language": "Java",
"level": "200",
"framework": "Terraform",
"introBox": {
"headline": "How it works",
"text": [
"Amazon API Gateway receives the HTTP POST request containing the file name to be uploaded",
"AWS Lambda is triggered by the API Gateway, this java based function generates Amazon S3 presigned URL.",
"A Java program is used to upload a file to the Amazon S3 bucket using the presigned URL"
]
},
"gitHub": {
"template": {
"repoURL": "https://github.com/aws-samples/serverless-patterns/tree/main/apigw-lambda-s3-terraform-java",
"templateURL": "serverless-patterns/apigw-lambda-s3-terraform-java",
"projectFolder": "apigw-lambda-s3-terraform-java",
"templateFile": "main.tf"
}
},
"resources": {
"bullets": [
{
"text": "Create a presigned URL for Amazon S3 using an AWS SDK",
"link": "https://docs.aws.amazon.com/AmazonS3/latest/userguide/example_s3_Scenario_PresignedUrl_section.html"
},
{
"text": "Uploading objects with presigned URLs",
"link": "https://docs.aws.amazon.com/AmazonS3/latest/userguide/PresignedUrlUploadObject.html"
}
]
},
"deploy": {
"text": [
"mvn clean package",
"terraform init",
"terraform apply"
]
},
"testing": {
"text": [
"See the GitHub repo for detailed testing instructions."
]
},
"cleanup": {
"text": [
"Delete the Amazon S3 bucket content: <code>aws s3 rm s3://{MyS3BucketName} --recursive</code>",
"Delete the stack: <code>terraform destroy</code>."
]
},
"authors": [
{
"name": "Jing Xiang Peh",
"image": "images/profile-pic.jpg",
"bio": "I am a Specialist Solutions Architect for Serveless based in Singapore",
"linkedin": "jing-xiang-peh"
}
],
"patternArch": {
"icon1": {
"x": 20,
"y": 50,
"service": "apigw",
"label": "Amazon API Gateway"
},
"icon2": {
"x": 50,
"y": 50,
"service": "lambda",
"label": "AWS Lambda"
},
"icon3": {
"x": 80,
"y": 50,
"service": "s3",
"label": "Amazon S3 Bucket"
},
"line1": {
"from": "icon1",
"to": "icon2",
"label": ""
},
"line2": {
"from": "icon2",
"to": "icon3",
"label": ""
}
}
}
61 changes: 61 additions & 0 deletions apigw-lambda-s3-terraform-java/example-pattern.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
{
"title": "Upload file to Amazon S3 bucket using presigned URL",
"description": "This project deploys Amazon API Gateway with AWS Lambda integration to showcase how to upload a file up to 5GB to Amazon S3 bucket using a presigned S3 URL.",
"language": "Java",
"level": "200",
"framework": "Terraform",
"introBox": {
"headline": "How it works",
"text": [
"Amazon API Gateway receives the HTTP POST request containing the file name to be uploaded",
"AWS Lambda is triggered by the API Gateway, this java based function generates Amazon S3 presigned URL.",
"A Java program is used to upload a file to the Amazon S3 bucket using the presigned URL"
]
},
"gitHub": {
"template": {
"repoURL": "https://github.com/aws-samples/serverless-patterns/tree/main/apigw-lambda-s3-terraform-java",
"templateURL": "serverless-patterns/apigw-lambda-s3-terraform-java",
"projectFolder": "apigw-lambda-s3-terraform-java",
"templateFile": "main.tf"
}
},
"resources": {
"bullets": [
{
"text": "Create a presigned URL for Amazon S3 using an AWS SDK",
"link": "https://docs.aws.amazon.com/AmazonS3/latest/userguide/example_s3_Scenario_PresignedUrl_section.html"
},
{
"text": "Uploading objects with presigned URLs",
"link": "https://docs.aws.amazon.com/AmazonS3/latest/userguide/PresignedUrlUploadObject.html"
}
]
},
"deploy": {
"text": [
"mvn clean package",
"terraform init",
"terraform apply"
]
},
"testing": {
"text": [
"See the GitHub repo for detailed testing instructions."
]
},
"cleanup": {
"text": [
"Delete the Amazon S3 bucket content: <code>aws s3 rm s3://{MyS3BucketName} --recursive</code>",
"Delete the stack: <code>terraform destroy</code>."
]
},
"authors": [
{
"name": "Jing Xiang Peh",
"image": "images/profile-pic.jpg",
"bio": "I am a Specialist Solutions Architect for Serveless based in Singapore",
"linkedin": "jing-xiang-peh"
}
]
}
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading

0 comments on commit cba5beb

Please sign in to comment.