adding a pre-commit hook to check for decrypted secrets (#707)

* adding a pre-commit hook to check for decrypted secrets

updating the pre-commit hook to compare the committed files to a
predefined lists, `.secrets`, indicating which files should be
encrypted.  If any committed file that has a match in `.secrets` appears
to be decrypted (aka in plaintext), the commit will fail

* adding initialize step to readme

Co-authored-by: John Zulim <[email protected]>
Co-authored-by: Ryan Travitz <[email protected]>

.gitignore

@@ -4,6 +4,7 @@
 !.gitignore
 !.gitattributes
 !.github
+!.secrets
 
 # OSX system files
 .DS_Store

.secrets

@@ -0,0 +1,17 @@
+ops/ansible/playbooks-ccs/host_vars/builds.bfd-mgmt.cmscloud.local/vault.yml
+ops/ansible/playbooks-ccs/files/prod-sbx-bluebutton-appserver-keystore.jks
+ops/ansible/playbooks-ccs/files/prod-bluebutton-appserver-keystore.jks
+ops/ansible/playbooks-ccs/files/client_data_server_local_test_env_test_keypair.pem
+ops/ansible/playbooks-ccs/files/client_data_server_local_test_env_prod_keypair.pem
+ops/ansible/playbooks-ccs/files/client_data_server_local_test_env_dpr_keypair.pem
+ops/ansible/playbooks-ccs/vars/000_cross_env_vars.yml
+ops/ansible/playbooks-ccs/vars/prod/group_vars/all/vault.yml
+ops/ansible/playbooks-ccs/vars/prod-sbx/group_vars/all/vault.yml
+ops/ansible/playbooks-ccs/files/test-bluebutton-appserver-keystore.jks
+ops/ansible/playbooks-ccs/vars/test/group_vars/all/vault.yml
+apps/bfd-server-test-perf/ansible/vars/prod.yml
+apps/bfd-server-test-perf/ansible/vars/test.yml
+apps/bfd-server-test-perf/ansible/vars/gdit.yml
+apps/bfd-server-test-perf/ansible/vaults/test.vault.yml
+apps/bfd-server-test-perf/ansible/vars/dpr.yml
+apps/bfd-server-test-perf/ansible/group_vars/all/vault.yml

README.md

@@ -66,10 +66,10 @@ mkdir -p ~/workspaces/bfd/
 git clone [email protected]:CMSgov/beneficiary-fhir-data.git ~/workspaces/bfd/beneficiary-fhir-data.git
 ```
 
-### Native Setup
+### Initializing the Repository
 1. Install JDK 8. You'll need Java 8 to run BFD. You can install OpenJDK 8 however you prefer.
-2. Install Maven 3. Project tasks are handled by Apache Maven. Install it however you prefer.
-3. Configure your toolchain. You'll want to configure your `~/.m2/toolchains.xml` file to look like the following (change the jdkHome appropriately):
+1. Install Maven 3. Project tasks are handled by Apache Maven. Install it however you prefer.
+1. Configure your toolchain. You'll want to configure your `~/.m2/toolchains.xml` file to look like the following (change the jdkHome appropriately):
     ```xml
     <?xml version="1.0" encoding="UTF8"?>
     <toolchains>
@@ -86,8 +86,11 @@ git clone [email protected]:CMSgov/beneficiary-fhir-data.git ~/workspaces/bfd/benef
       </toolchain>
     </toolchains>
     ```
-4. Change to the `apps/` directory and `mvn clean install -DskipITs`. The flag to skip the integration tests is important here. You will need to have AWS access for the integration tests to work correctly.
-5. Set up a Postgres 12 database with the following command. Data will be persisted between starts and stops in the `bfd_pgdata` volume.
+1. Install pre-commit hooks `mvn -f apps initialize`
+
+### Native Setup
+1. Change to the `apps/` directory and `mvn clean install -DskipITs`. The flag to skip the integration tests is important here. You will need to have AWS access for the integration tests to work correctly.
+1. Set up a Postgres 12 database with the following command. Data will be persisted between starts and stops in the `bfd_pgdata` volume.
     ```sh
     docker run \
       -d \
@@ -99,26 +102,26 @@ git clone [email protected]:CMSgov/beneficiary-fhir-data.git ~/workspaces/bfd/benef
       -v 'bfd_pgdata:/var/lib/postgresql/data' \
       postgres:12
     ```
-6. To load one test beneficiary, with your database running, change directories into `apps/bfd-pipeline/bfd-pipeline-ccw-rif` and run:
+1. To load one test beneficiary, with your database running, change directories into `apps/bfd-pipeline/bfd-pipeline-ccw-rif` and run:
     ```
     mvn -Dits.db.url="jdbc:postgresql://localhost:5432/bfd" -Dits.db.username=bfd -Dits.db.password=InsecureLocalDev -Dit.test=RifLoaderIT#loadSampleA clean verify
     ```
     This will kick off the integration test `loadSampleA`. After the job completes, you can verify that it ran properly with:
     ```
     docker exec bfd-db psql 'postgresql://bfd:InsecureLocalDev@localhost:5432/bfd' -c 'SELECT "beneficiaryId" FROM "Beneficiaries" LIMIT 1;'
     ```
-7. Run `export BFD_PORT=6500`. The actual port is not important, but without it the `start-server` script will pick a different one each time, which gets annoying later. This can be set in your shell profile but note that when running the integration tests through maven, the BFD_PORT needs to be unset from the environment.
-8. Now it's time to start the server up. Change to `apps/bfd-server` and run:
+1. Run `export BFD_PORT=6500`. The actual port is not important, but without it the `start-server` script will pick a different one each time, which gets annoying later. This can be set in your shell profile but note that when running the integration tests through maven, the BFD_PORT needs to be unset from the environment.
+1. Now it's time to start the server up. Change to `apps/bfd-server` and run:
     ```
     mvn -Dits.db.url="jdbc:postgresql://localhost:5432/bfd?user=bfd&password=InsecureLocalDev" --projects bfd-server-war package dependency:copy antrun:run org.codehaus.mojo:exec-maven-plugin:exec@server-start
     ```
     After it starts up, you can tail the logs with `tail -f bfd-server-war/target/server-work/server-console.log`
-9. We're finally going to make a request. BFD requires that clients authenticate themselves with a certificate. Those certs live in the `apps/bfd-server/dev/ssl-stores` directory. We can curl the server using a cert with this command:
+1. We're finally going to make a request. BFD requires that clients authenticate themselves with a certificate. Those certs live in the `apps/bfd-server/dev/ssl-stores` directory. We can curl the server using a cert with this command:
     ```
     curl --silent --insecure --cert $BFD_PATH/apps/bfd-server/dev/ssl-stores/client-unsecured.pem "https://localhost:$BFD_PORT/v2/fhir/ExplanationOfBenefit/?patient=567834&_format=json"
     ```
     where `$BFD_PATH` is that path to the `beneficiary-fhir-data` repo on your system. It may be helpful to have that set in your profile, too. To configure Postman, go to `Settings -> Certificates -> Add certificate` and load in `apps/bfd-server/dev/ssl-stores/client-trusted-keystore.pfx` under the PFX File option. The passphrase is `changeit`. Under `Settings -> General` you'll also want to turn off "SSL Certificate Verification."
-10. Total success (probably)!. You have a working call. To stop the server run this from the `apps/bfd-server` directory:
+1. Total success (probably)!. You have a working call. To stop the server run this from the `apps/bfd-server` directory:
     ```
     mvn -Dits.db.url="jdbc:postgresql://localhost:5432/bfd?user=bfd&password=InsecureLocalDev" --projects bfd-server-war package dependency:copy antrun:run org.codehaus.mojo:exec-maven-plugin:exec@server-stop
     ```

apps/pom.xml

@@ -319,7 +319,7 @@
 					<configuration>
 						<hooks>
 							<pre-commit>
-#!/bin/sh
+#!/usr/bin/env bash
 set -e
 
 checkJavaFormat() {
@@ -346,6 +346,37 @@ checkJavaFormat() {
   return ${checkResult}
 }
 
+checkSecretFilesForPlainText() {
+  echo 'Verifying secret files are not in plaintext...'
+  set +e
+  # read list of files containing secrets
+  IFS=$'\n' read -d '' -r -a secrets &lt; .secrets
+
+  # make list of files to be committed by printing out a newline
+  # separated list of files staged for commit to a temporary file
+  # and reading it as an array
+  tmpfile=$(mktemp)
+  git diff --cached --name-only --diff-filter=ACM > $tmpfile
+  IFS=$'\n' read -d '' -r -a commits &lt; $tmpfile
+
+  # for all files to be committed which are a secret file,
+  # grep for the header "$ANSIBLE_VAULT;1.1;AES256"
+  # if it's not there, abort.
+  for commitFile in ${commits[@]}; do
+    for secretFile in ${secrets[@]}; do
+      if [ $commitFile == $secretFile ]; then
+        header=$(echo "$(git show :./$secretFile)" | grep "\$ANSIBLE_VAULT;1.1;AES256")
+        if [ -z $header ]; then
+          echo "attempting to commit an unencrypted secret: $secretFile; aborting"
+          exit 1
+        fi
+      fi
+    done
+  done
+  set -e
+}
+
+checkSecretFilesForPlainText
 checkJavaFormat
 							</pre-commit>
 						</hooks>