rawls

1. The workspace/entity/submission/method config manager for Terra
2. Bill Rawls, Deputy Commissioner for Operations from The Wire:

Getting started

git clone [email protected]:broadinstitute/rawls.git
cd rawls
sbt antlr4:antlr4Generate # Generates source code for IntellIJ IDEA

scalafmt

To format files, scalafmt can be set up to run from Intellij

When a PR is opened, scalafmt will check formatting for modified files.

To run scalafmt from the commandline:

sbt scalafmt

Unit Testing with MySQL in Docker

Ensure that docker is up to date and initialized. Spin up mysql locally and validate that it is working:

./docker/run-mysql.sh start

Run tests.

export JAVA_OPTS="-Xmx2G -Xms1G -Dmysql.host=localhost -Dmysql.port=3310"
sbt clean compile test

And when you're done, spin down mysql (it is also fine to leave it running for your next round of tests):

./docker/run-mysql.sh stop

Running Locally

Requirements

• Broad campus network or NonSplit VPN
• Docker Desktop (8GB+ recommended)
• kubectl and gcloud installed.

Initial setup

1. You will need to authenticate in gcloud; if you are not already then running the script will ask you to.
2. Render the local configuration files. From the root of the Rawls repo, run:

./local-dev/bin/render

3. Edit the /etc/hosts file on your machine to add this entry for calling Rawls endpoints:

127.0.0.1	local.dsde-dev.broadinstitute.org

Front Rawls (default)

Next, execute the following command from the root of the Rawls repo:

# Requires Broad campus network or NonSplit VPN
./config/docker-rsync-local-rawls.sh

By default, this will set up an instance of rawls pointing to the database and Sam in dev. It will also set up a process that will watch the local files for changes, and restart the service when source files change.

See docker-rsync-local-rawls.sh for more configuration options.

When Rawls starts up, access the Rawls Swagger page: https://local.dsde-dev.broadinstitute.org:20443/

Back Rawls

Additional setup for Back Rawls

1. Personal clone of the Rawls Dev database
2. Edit local-dev/templates/sqlproxy.env to set your clone instance name
3. Re-render local config: ./local-dev/bin/render

By default, a locally run Rawls will boot as a "front" instance of Rawls. A front Rawls will serve all HTTP requests and can modify the database, but it will not do monitoring tasks such as submission monitoring, PFB imports, or Google billing project creation.

If you are developing a ticket that deals with any sort of monitoring or asynchronous features, you will likely want to boot your Rawls as a "back" instance, which will run a fully-featured instance of Rawls with monitoring tasks enabled. To boot your local instance as a "back" instance, run:

Likewise, use a DB clone whenever writing Liquibase migrations or doing database work, so as not to disrupt the shared Dev instance.

# Requires Broad campus network or NonSplit VPN
BACK_RAWLS=true ./config/docker-rsync-local-rawls.sh

Developer quick links:

• Swagger UI: https://rawls.dsde-dev.broadinstitute.org

Build Rawls docker image

Build Rawls jar

./docker/build.sh jar

Build Rawls jar and docker image

./docker/build.sh jar -d build

Publish rawls-model

Supported Scala versions: 2.13

Running the publishRelease.sh script publishes a release of rawls-model, workbench-util and workbench-google to Artifactory. You should do this manually from the base directory of the repo when you change something in model/src, util/src or google/src.

Note: We have started just using the automatically generated -SNAP versions published by rawls-build GitHub action on every dev build. Here are detailed instructions for finding the name of the jar file:

• Navigate to the rawls-build-tag-publish-and-run-tests workflow github action for your commit
• Navigate to the rawls-build-publish-job job of that workflow
• Open the "dispatch build" step, and click over to the run in terra-github-workflows
• Expand the "Publish model library" step
• Note that version published, e.g. "rawls-model_2.13-v0.0.180-SNAP.jar"

To publish a temporary or test version, use publishSnapshot.sh like so:

VAULT_TOKEN=$(cat ~/.vault-token) ARTIFACTORY_USERNAME=dsdejenkins ARTIFACTORY_PASSWORD=$(docker run -e VAULT_TOKEN=$VAULT_TOKEN broadinstitute/dsde-toolbox:dev vault read -field=password secret/dsp/accts/artifactory/dsdejenkins) core/src/bin/publishSnapshot.sh

To publish an official release, you can run the following command:

VAULT_TOKEN=$(cat ~/.vault-token) ARTIFACTORY_USERNAME=dsdejenkins ARTIFACTORY_PASSWORD=$(docker run -e VAULT_TOKEN=$VAULT_TOKEN broadinstitute/dsde-toolbox:dev vault read -field=password secret/dsp/accts/artifactory/dsdejenkins) core/src/bin/publishRelease.sh

You can view what is in the artifactory here: https://broadinstitute.jfrog.io/broadinstitute/webapp/#/home

After publishing:

• Update model/CHANGELOG.md properly
• Update the rawls-model dependency in the automation subdirectory, and ensure that the sbt project is healthy
• In workbench-libs, update the rawls-model dependency in the project dependencies, update the README and the serviceTest changelog accordingly, and ensure that the sbt project is healthy
• In firecloud-orchestration, update the rawls-model dependency in the project dependencies, and ensure that the sbt project is healthy

Troubleshooting

If you get the error message release version 17 not supported:

• Run java -version and verify that you're running 17. If not, you will need to install / update your PATH.

For integration test issues, see automation/README.md.

Debugging in Intellij IDEA

You can attach Intellij's interactive debugger to Rawls running locally in a docker container run via the ./config/docker-rsync-local-rawls.sh script.

Add a "Remote JVM Debug" configuration that attaches to localhost on port 25050. See the link below for more detailed steps. https://blog.jetbrains.com/idea/2019/04/debug-your-java-applications-in-docker-using-intellij-idea/