Skip to content

Commit

Permalink
update documentation
Browse files Browse the repository at this point in the history
  • Loading branch information
James Hutchby committed Oct 25, 2022
1 parent dd481d8 commit 83d066c
Showing 1 changed file with 74 additions and 63 deletions.
137 changes: 74 additions & 63 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,33 +10,43 @@ It can be configured to run periodically using CloudWatch events.
## Quick start

1. Create an AWS lambda function:
- Author from scratch
- Runtime: Node.js 14.x
- Author from scratch
- Runtime: Node.js 16.x
- Architecture: x86_64
2. tab "Code" -> "Upload from" -> ".zip file":
- Upload ([pgdump-aws-lambda.zip](https://github.com/jameshy/pgdump-aws-lambda/releases/latest))
- tab "Configuration" -> "General Configuration" -> "Edit"
- Timeout: 15 minutes
- Edit the role and attach the policy "AmazonS3FullAccess"
- Save
3. Test
- Create new test event, e.g.:
```json
{
"PGDATABASE": "dbname",
"PGUSER": "postgres",
"PGPASSWORD": "password",
"PGHOST": "host",
"S3_BUCKET" : "db-backups",
"ROOT": "hourly-backups"
}
```
- *Test* and check the output

4. Create a CloudWatch rule:
- Event Source: Schedule -> Fixed rate of 1 hour
- Targets: Lambda Function (the one created in step #1)
- Configure input -> Constant (JSON text) and paste your config (as per previous step)
- Upload ([pgdump-aws-lambda.zip](https://github.com/jameshy/pgdump-aws-lambda/releases/latest))
- tab "Configuration" -> "General Configuration" -> "Edit"
- Timeout: 15 minutes
- Edit the role and attach the policy "AmazonS3FullAccess"
- Save
3. Give your lambda permissions permissions to write to S3:

- tab "Configuration" -> "Permissions"
- click the existing Execution role
- "Add permissions" -> "Attach policies"
- select "AmazonS3FullAccess" and click "Attach policies"

4. Test

- Create new test event, e.g.:

```json
{
"PGDATABASE": "dbname",
"PGUSER": "postgres",
"PGPASSWORD": "password",
"PGHOST": "host",
"S3_BUCKET": "db-backups",
"ROOT": "hourly-backups"
}
```

- _Test_ and check the output

5. Create a CloudWatch rule:
- Event Source: Schedule -> Fixed rate of 1 hour
- Targets: Lambda Function (the one created in step #1)
- Configure input -> Constant (JSON text) and paste your config (as per previous step)

#### File Naming

Expand All @@ -55,13 +65,13 @@ You can add an encryption key to your event, e.g.

```json
{
"PGDATABASE": "dbname",
"PGUSER": "postgres",
"PGPASSWORD": "password",
"PGHOST": "host",
"S3_BUCKET" : "db-backups",
"ROOT": "hourly-backups",
"ENCRYPT_KEY": "c0d71d7ae094bdde1ef60db8503079ce615e71644133dc22e9686dc7216de8d0"
"PGDATABASE": "postgres",
"PGUSER": "postgres",
"PGPASSWORD": "password",
"PGHOST": "host",
"S3_BUCKET": "db-backups",
"ROOT": "hourly-backups",
"ENCRYPT_KEY": "c0d71d7ae094bdde1ef60db8503079ce615e71644133dc22e9686dc7216de8d0"
}
```

Expand All @@ -88,14 +98,13 @@ Your context may require that you use IAM-based authentication to log into the P
Support for this can be enabled my making your Cloudwatch Event look like this.

```json

{
"PGDATABASE": "dbname",
"PGUSER": "postgres",
"PGHOST": "host",
"S3_BUCKET" : "db-backups",
"ROOT": "hourly-backups",
"USE_IAM_AUTH": true
"PGDATABASE": "dbname",
"PGUSER": "postgres",
"PGHOST": "host",
"S3_BUCKET": "db-backups",
"ROOT": "hourly-backups",
"USE_IAM_AUTH": true
}
```

Expand All @@ -111,67 +120,69 @@ NOTE: the execution role for the Lambda function must have access to GetSecretVa
Support for this can be enabled by setting the SECRETS_MANAGER_SECRET_ID, so your Cloudwatch Event looks like this:

```json

{
"SECRETS_MANAGER_SECRET_ID": "my/secret/id",
"S3_BUCKET" : "db-backups",
"ROOT": "hourly-backups"
"SECRETS_MANAGER_SECRET_ID": "my/secret/id",
"S3_BUCKET": "db-backups",
"ROOT": "hourly-backups"
}
```

If you supply `SECRETS_MANAGER_SECRET_ID`, you can ommit the 'PG*' keys, and they will be fetched from your SecretsManager secret value instead with the following mapping:
If you supply `SECRETS_MANAGER_SECRET_ID`, you can ommit the 'PG\*' keys, and they will be fetched from your SecretsManager secret value instead with the following mapping:

| Secret Value | PG-Key |
| ------------- | ------------- |
| username | PGUSER |
| password | PGPASSWORD |
| dbname | PGDATABASE |
| host | PGHOST |
| port | PGPORT |
| Secret Value | PG-Key |
| ------------ | ---------- |
| username | PGUSER |
| password | PGPASSWORD |
| dbname | PGDATABASE |
| host | PGHOST |
| port | PGPORT |


You can provide overrides in your event to any PG* keys as event parameters will take precedence over secret values.
You can provide overrides in your event to any PG\* keys as event parameters will take precedence over secret values.

## Developer

#### Bundling a new `pg_dump` binary

1. Launch an EC2 instance with the Amazon Linux 2 AMI
2. Connect via SSH and:

```bash

# install postgres 13
# install postgres 15
sudo amazon-linux-extras install epel

sudo tee /etc/yum.repos.d/pgdg.repo<<EOF
[pgdg13]
name=PostgreSQL 13 for RHEL/CentOS 7 - x86_64
baseurl=https://download.postgresql.org/pub/repos/yum/13/redhat/rhel-7-x86_64
[pgdg15]
name=PostgreSQL 15 for RHEL/CentOS 7 - x86_64
baseurl=https://download.postgresql.org/pub/repos/yum/15/redhat/rhel-7-x86_64
enabled=1
gpgcheck=0
EOF

sudo yum install postgresql13 postgresql13-server
sudo yum install postgresql15 postgresql15-server

exit
```

#### Download the binaries

```bash
scp -i ~/aws.pem ec2-user@18.157.84.236:/usr/bin/pg_dump ./bin/postgres-13.3/pg_dump
scp -i ~/aws.pem ec2-user@18.157.84.236:/usr/lib64/{libcrypt.so.1,libnss3.so,libsmime3.so,libssl3.so,libsasl2.so.3,liblber-2.4.so.2,libldap_r-2.4.so.2} ./bin/postgres-13.3/
scp -i ~/aws.pem ec2-user@18.157.84.236:/usr/pgsql-13/lib/libpq.so.5 ./bin/postgres-13.3/libpq.so.5
scp ec2-user@ec2-18-194-222-2.eu-central-1.compute.amazonaws.com:/usr/bin/pg_dump ./bin/postgres-15.0/pg_dump
scp ec2-user@ec2-18-194-222-2.eu-central-1.compute.amazonaws.com:/usr/lib64/{libcrypt.so.1,libnss3.so,libsmime3.so,libssl3.so,libsasl2.so.3,liblber-2.4.so.2,libldap_r-2.4.so.2} ./bin/postgres-15.0/
scp ec2-user@ec2-18-194-222-2.eu-central-1.compute.amazonaws.com:/usr/pgsql-15/lib/libpq.so.5 ./bin/postgres-15.0/libpq.so.5
```

3. To use the new postgres binary pass PGDUMP_PATH in the event:

```json
{
"PGDUMP_PATH": "bin/postgres-13.3"
"PGDUMP_PATH": "bin/postgres-15.0"
}
```

#### Creating a new function zip

`npm run deploy`
`npm run makezip`

#### Contributing

Expand Down

0 comments on commit 83d066c

Please sign in to comment.