Merge pull request #125 from pangeo-forge/use-aws-kms

Use AWS KMS for SOPS encryption

.prettierignore

@@ -0,0 +1 @@
+secrets/*

.sops.yaml

@@ -0,0 +1,5 @@
+# See https://github.com/mozilla/sops#using-sops-yaml-conf-to-select-kms-pgp-for-new-files
+
+creation_rules:
+  - path_regex: secrets/.*
+    kms: 'arn:aws:kms:us-east-1:256317687910:key/d8b153c3-20a9-4364-a553-94405d4c1027,arn:aws:kms:us-west-1:256317687910:key/0f31de65-bcc9-4fef-b9f5-67ce086f532e'

Dockerfile

@@ -28,6 +28,11 @@ RUN wget --quiet https://releases.hashicorp.com/terraform/${TF_VERSION}/terrafor
     && mv terraform /usr/local/bin/terraform \
     && rm terraform_${TF_VERSION}_linux_amd64.zip
 
+# Install aws cli, for decrypting secrets via AWS KMS
+RUN curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" \
+    && unzip awscliv2.zip \
+    && ./aws/install
+
 # Install gcloud https://cloud.google.com/sdk/docs/install#installation_instructions
 RUN echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt \
     cloud-sdk main" | tee -a /etc/apt/sources.list.d/google-cloud-sdk.list \

app.json

@@ -13,7 +13,7 @@
   "environments": {
     "review": {
       "env": {
-        "PANGEO_FORGE_DEPLOYMENT": "pforge-pr-80"
+        "PANGEO_FORGE_DEPLOYMENT": "pforge-pr-125"
       }
     }
   }

docker-compose.yml

@@ -12,18 +12,12 @@ services:
       - '3000:8000'
     depends_on:
       - db
-    volumes:
-      # See SOPS_AGE_KEY_FILE in `environment:` below for details.
-      - ${PWD}/key.txt:/var/sops/key.txt
     environment:
       # Nearly the same config as for postgres service in `.github/workflows/main.yaml`,
       # with exception of hostname. For hostname, see https://docs.docker.com/compose/networking/
       DATABASE_URL: 'postgres://postgres_user:postgres_password@db:5432/postgres_db'
       # TODO: variablize this so others can use with their own local config
       PANGEO_FORGE_DEPLOYMENT: 'pforge-local-cisaacstern'
-      # In Heroku, we use the SOPS_AGE_KEY variable instead, for description of both options, see:
-      # https://github.com/mozilla/sops#22encrypting-using-age
-      SOPS_AGE_KEY_FILE: '/var/sops/key.txt'
       # Get the project_id dynamically after dataflow key file is decrypted in entrypoint.
       # Python command for parsing JSON is set as env var because putting it directly in
       # entrypoint is too many layers of nested quotation marks.
@@ -42,6 +36,8 @@ services:
     entrypoint: >
       /bin/sh -c 'sleep 1
       && python3.9 -m alembic upgrade head
+      && aws configure set aws_access_key_id $AWS_ACCESS_KEY_ID
+      && aws configure set aws_secret_access_key $AWS_SECRET_ACCESS_KEY
       && sops -d -i ./secrets/bakery-args.pangeo-ldeo-nsf-earthcube.yaml
       && sops -d -i ./secrets/config.$${PANGEO_FORGE_DEPLOYMENT}.yaml
       && sops -d -i $${DATAFLOW_CREDS}

docs/development_guide.md

@@ -11,8 +11,8 @@ through making your first PR to `pangeo-forge-orchestrator`.
     - [2.1.1 GitHub App](#211-github-app)
     - [2.1.2 FastAPI](#212-fastapi)
   - [2.2 Encrypting & committing creds](#22-encrypting--committing-creds)
-    - [2.2.1 Create an age key](#221-create-an-age-key)
-    - [2.2.2 Encrypt those creds](#222-encrypt-those-creds)
+    - [2.2.1 pre-commit-hook-ensure-sops](#221-pre-commit-hook-ensure-sops)
+    - [2.2.2 Encryption](#222-encryption)
   - [2.3 Database](#23-database)
   - [2.4 The proxy: selection & setup](#24-the-proxy-selection--setup)
     - [2.4.1 smee vs. ngrok](#241-smee-vs-ngrok)
@@ -235,10 +235,12 @@ are for your local dev environment), this is a good opportunity to practice encr
 _will_ be required for the `review` deployment later on). Moreover, managing all credentials (including for
 `local` deployment) in a uniform manner simplifies the process.
 
-> ⚠️ If you have not yet made sure that [**pre-commit is installed**](https://pre-commit.com/#quick-start) in
-> your local development environment, now is the time to do so! If you do not install pre-commit,
-> [pre-commit-hook-ensure-sops](https://github.com/yuvipanda/pre-commit-hook-ensure-sops) cannot protect you
-> from accidentally committing unencrypted credentials.
+#### 2.2.1 pre-commit-hook-ensure-sops
+
+⚠️ If you have not yet made sure that [**pre-commit is installed**](https://pre-commit.com/#quick-start) in
+your local development environment, now is the time to do so! If you do not install pre-commit,
+[pre-commit-hook-ensure-sops](https://github.com/yuvipanda/pre-commit-hook-ensure-sops) cannot protect you
+from accidentally committing unencrypted credentials.
 
 Assuming you have pre-commit installed (really, ☝️ read the warning above if you haven't already 😄), you will
 be protected from committing your (currently unencrypted) credentials:
@@ -252,35 +254,14 @@ Ensure secrets are encrypted with sops...................................Failed
 secrets/config.local.yaml: sops metadata key not found in file, is not properly encrypted
 ```
 
-#### 2.2.1 Create an age key
-
-Currently, we use the [`age` backend](https://github.com/mozilla/sops#encrypting-using-age) for SOPS.
-To get an `age` encryption key:
-
-1. [Install `age`](https://github.com/FiloSottile/age#installation)
-2. Generate your private/public key pair with `age-keygen -o key.txt`
-3. Your public key will be printed to stdout, and your private key will be saved to `key.txt`
-
-You can keep your private `key.txt` in the repo root; our `.gitignore` prevents it from being committed. Save
-your public key somewhere you'll remember it, and also export it to your local env as `AGE_PUBLIC_KEY`.
-For example:
-
-```console
-$ export AGE_PUBLIC_KEY=age1xvhtwvxklw0zljmj4flnqyqlnq0cudadnenzcv20snwa90nzqylqzqtlsg
-```
+#### 2.2.2 Encryption
 
-#### 2.2.2 Encrypt those creds
+> **Note**: From this point forward, you will need be a member of the `pangeo-forge` AWS project
+> with KMS permissions, and be logged in via `aws configure`.
 
 1. Install [SOPS](https://github.com/mozilla/sops). The easiest way to do this on Mac is probably
    [`brew install sops`](https://formulae.brew.sh/formula/sops).
-2. Set the `SOPS_AGE_RECIPIENTS` env var as follows:
-   ```console
-   $ export SOPS_AGE_RECIPIENTS=$(cat age-recipients.txt),${AGE_PUBLIC_KEY}
-   ```
-   This will ensure that you will be able to decrypt this file (because `AGE_PUBLIC_KEY` is your public key),
-   and that Pangeo Forge admins (whose public key is stored in `age-recipients.txt`) will also be able to
-   decrypt it.
-3. From the repo root, run:
+2. From the repo root, run:
    ```console
    $ sops -e -i secrets/config.local.yaml
    ```
@@ -430,7 +411,7 @@ Running the next script will require your GitHub App credentials, which if you'v
 sequentially are still encrypted. To decrypt them, from the repo root, run:
 
 ```console
-$ SOPS_AGE_KEY=$(cat key.txt) sops -d -i secrets/config.local.yaml
+$ sops -d -i secrets/config.local.yaml
 ```
 
 Then, using the proxy url you generated in [section 2.4.1](#241-smee-vs-ngrok) as `PROXY_URL`, run:
@@ -456,7 +437,6 @@ To recap, if you've followed this document sequentially, you should now have:
 
 <p style="margin-left:1.5em">
 ☑️ A <code>secrets/config.local.yaml</code> file with creds for a GitHub App & for FastAPI<br>
-☑️ An age private/public key pair, and the ability to use it, via SOPS, to encrypt and decrypt your creds<br>
 ☑️ pre-commit installed, to prevent you from accidentally committing unencrypted secrets<br>
 ☑️ <code>DATABASE_URL</code> env variable set<br>
 ☑️ Proxy client running in a terminal window<br>
@@ -472,7 +452,7 @@ In addition to the pre-requistes listed in the [recap](#25-interlude-recap) abov
 1. Your creds are decrypted:
 
    ```console
-   $ SOPS_AGE_KEY=$(cat key.txt) sops -d -i secrets/config.local.yaml
+   $ sops -d -i secrets/config.local.yaml
    ```
 
 2. You have `pangeo-forge-orchestrator` installed:
@@ -812,8 +792,10 @@ https://pangeo-forge-api-pr-80.herokuapp.com will be broken
    $ python3 scripts/generate_api_key.py review
    ```
 4. Encrypt the creds:
-   `sops -e -i secrets/config.review.yaml` > Assuming you are in the same terminal session, your `SOPS_AGE_RECIPIENTS` env variable will still
-   be set. If it's not, revisit [section 2.2.2](#222-encrypt-those-creds).
+
+   ```
+   sops -e -i secrets/config.review.yaml
+   ```
 
 5. Commit the encrypted `secrets/config.review.yaml` and push it to your PR branch.
 

scripts.deploy/release.sh

@@ -35,6 +35,10 @@ else:
 export TF_ENV=$(python3.9 -c "${SET_TF_ENV}")
 echo "terraform env set to '${TF_ENV}'"
 
+echo "setting aws config for kms access..."
+aws configure set aws_access_key_id $AWS_ACCESS_KEY_ID
+aws configure set aws_secret_access_key $AWS_SECRET_ACCESS_KEY
+
 echo "decrypting app secrets..."
 # For the persistent deployments 'pangeo-forge' & 'pangeo-forge-staging', APP_NAMES will always
 # be a single-element array, in which the sole value is identical to TF_ENV.
@@ -86,14 +90,6 @@ terraform -chdir='./terraform/'${TF_ENV} plan -out tfplan \
 terraform -chdir='./terraform/'${TF_ENV} apply tfplan
 
 echo "re-encrypting secrets..."
-# if AGE_PUBLIC_KEY is set, include it in encryption recipients.
-# this is never set in a real release, but is useful for working with this script locally.
-if [[ -z "${AGE_PUBLIC_KEY}" ]]; then
-    export SOPS_AGE_RECIPIENTS=$(cat age-recipients.txt)
-else
-    export SOPS_AGE_RECIPIENTS=$(cat age-recipients.txt),${AGE_PUBLIC_KEY}
-fi
-
 echo "re-encrypting terraform secrets..."
 sops -e -i "./${TF_CREDS}"
 for app in ${app_array[@]}; do

secrets/bakery-args.pangeo-ldeo-nsf-earthcube.yaml

@@ -1,33 +1,23 @@
 TargetStorage:
-  fsspec_args:
-    key: ENC[AES256_GCM,data:m98Vh1tNGK0wUY9pOIBkfMG4TlQ=,iv:+MxM8Kjjens9b7JZcvvy4KzahGpW8o7JG3ulvdyHsHU=,tag:SvnJAHCSKxzmsPMcgH2ezQ==,type:str]
-    secret: ENC[AES256_GCM,data:V0jtqFpLO3kGykTcCtZFaNSxLrkRKa5qGbMn1wZxZFxYYkZiDr84eQ==,iv:af2BPWRs2I3jJ6ML/+MrgJU8gUiQXFLLTwcpcoWYRG0=,tag:udQu/8+4sReMgM7lBCMw+A==,type:str]
+    fsspec_args:
+        key: ENC[AES256_GCM,data:APzcUozWDU5ITcdBho9OEJcCTRs=,iv:z7iyaHJ7QYu4R2quVycfZpVDVhsQHiilAfimr8RSSq0=,tag:J/+XEYa+8OLXNfr146G0BA==,type:str]
+        secret: ENC[AES256_GCM,data:JdMxI4tkaI1yIYL+zuzhoCB7qHXxsnRROb3XusecpX+fK781gtF3mg==,iv:Q5Y+7qyV42OZA1eHEQaOse8Q1HPDBx5TVMBU8FoOKCg=,tag:fEDqgm70284thqf/jjvkYg==,type:str]
 sops:
-  kms: []
-  gcp_kms: []
-  azure_kv: []
-  hc_vault: []
-  age:
-    - recipient: age14cdv6w5wqfpw2eha9z436zy02nqhg5xwrava0clh2yad84r5238qkawfnx
-      enc: |
-        -----BEGIN AGE ENCRYPTED FILE-----
-        YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBROEdUNExWb2RhVG5Bd0Ja
-        TTFBUWVUK3cwMVFRcjZrVkZ0dkVaZ05VNEZzCkQySEREYnpOV250cU5IWFNOdDRT
-        L2VEZjhCMGJYOG9maXp2MzAvdnZtN3cKLS0tIGpQUml3Wkw5cE9ITWZnN1VsREhm
-        RmxBUmlmYi9QeW1HZ25YR1pWV0t5ajQKqOgUxi4oIem6qAFGP60b0l7H8ExY3jVe
-        lN+kxB3QxMuvr6FzyHr7B8UlDqdQwiBKGXVHXsa6pgX4pQ2FuDELqA==
-        -----END AGE ENCRYPTED FILE-----
-    - recipient: age1s4x6q59pfwkeq2hu85zfhpspar25484ac5fzulvr0jf5j6ayqq2sjyw2yg
-      enc: |
-        -----BEGIN AGE ENCRYPTED FILE-----
-        YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBHL3ZYWnVUd0EwMUFMd1Vm
-        b2pYdFNjem9oZGtYR0RZWCtSamFrYWd4M1FZCkFoN1VpMWQ5SkNwYzlZaWw3RTFt
-        Qm1GT1V6TXJWbHhiY1BEbWliWThSa3MKLS0tIG1mQiswYjBHdWpBRXcySWpiUGpv
-        S2RhN05IZkFWUzQxempVeXZzc1hKejAKi19uhx8PTi2tcJ5q5otvF7AcaAWFJz5V
-        2kbgK96Ha+Qsbw5faboP5W243iSUNxv9Ys3iDrapriYF2aCa/JZ0Sw==
-        -----END AGE ENCRYPTED FILE-----
-  lastmodified: '2022-09-02T18:42:02Z'
-  mac: ENC[AES256_GCM,data:MJwXrLlkKxb0mR1/n6o9zVyzQrXTvOM6aCBpIe4KeCK8PShnohSr3+QElyWwo6PYjFOq4HyTgFObxlByy2fppJTzRASz9Vvvh2NB/b3d3VhGeS0kZjGMjBfNs9fHFA9rZKCGpQQ81DRL2hOuOOHo5vC1n/t5rfjida0dyhyzq2w=,iv:bdqiZi6mZ/NnoMprWYodiWUIIuwgTkh5WWCoGwVRyIg=,tag:68wa+VPD3X6xEpTyWWHsjA==,type:str]
-  pgp: []
-  unencrypted_suffix: _unencrypted
-  version: 3.7.3
+    kms:
+        - arn: arn:aws:kms:us-east-1:256317687910:key/d8b153c3-20a9-4364-a553-94405d4c1027
+          created_at: "2022-09-15T00:26:36Z"
+          enc: AQICAHgpH4G+b2ULBcvMucaHVvQi3QdX1B0xlVvF3iFfhxUVOwGBxwL5U+LK18IsadGDA/d8AAAAfjB8BgkqhkiG9w0BBwagbzBtAgEAMGgGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQM8k58KV6L3/UCn2VLAgEQgDuFF8Kw2PGvL0Wuua0uitbNZQjqOCdTbL94ZIqV0NeFsV/amjjQJRoZtUZaICYfHigN7x4ROe14xWA4kA==
+          aws_profile: ""
+        - arn: arn:aws:kms:us-west-1:256317687910:key/0f31de65-bcc9-4fef-b9f5-67ce086f532e
+          created_at: "2022-09-15T00:26:36Z"
+          enc: AQICAHiYQfpCkhiXnfU8apZYPITKv6caFhMAr7Hx04ufaTWvvgGSiaija70ObeE+rcP//gfbAAAAfjB8BgkqhkiG9w0BBwagbzBtAgEAMGgGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQMmRv56PDz5+cPV/sMAgEQgDsWE9TKunRAn9HV3LKwEswxfzOr2SddqaF1zB59n51r4Tsrs0Ypzg14eurUc8Dc9cHAvoQXml+GEZ8fDw==
+          aws_profile: ""
+    gcp_kms: []
+    azure_kv: []
+    hc_vault: []
+    age: []
+    lastmodified: "2022-09-15T00:26:37Z"
+    mac: ENC[AES256_GCM,data:5XeqjWY/4+HPTBY0IyoHtnPnTrX/AnrfhG10CPUGETcxoyCcWxPBOnGdm5/HGvDQQAhZM9Z+senD7ov1c0M0MG4WF7IINVjojk0lPFoYS9ajLMWzDDkq1kLALkrKMHroIlq79Fkk/ojd7jCPxTcHfJsTY2QurAh0eKaBNxpI/50=,iv:xVRF4dHde/ccNndu0bZ32fu5nNt8/8R7jv+BieWQ7RE=,tag:edtixXTnwcaGl1N3h2KyvQ==,type:str]
+    pgp: []
+    unencrypted_suffix: _unencrypted
+    version: 3.7.3