[AN-226] Enable submitting workflows to Cromwell's GCP Batch backend

.github/PULL_REQUEST_TEMPLATE.md

@@ -8,7 +8,10 @@ Ticket: <Link to Jira ticket>
 - [ ] Include the JIRA issue number in the PR description and title
 - [ ] Make sure Swagger is updated if API changes
   - [ ] **...and Orchestration's Swagger too!**
-- [ ] If you changed anything in `model/`, then you should [publish a new official `rawls-model`](https://github.com/broadinstitute/rawls/blob/develop/README.md#publish-rawls-model) and update `rawls-model` in [Orchestration's dependencies](https://github.com/broadinstitute/firecloud-orchestration/blob/develop/project/Dependencies.scala).
+- [ ] If you changed anything in `model/`, then you should publish a new official `rawls-model` and perform the corresponding dependency updates as specified in the [README](https://github.com/broadinstitute/rawls/blob/develop/README.md#publish-rawls-model):
+  - [ ] in the automation subdirectory
+  - [ ] in workbench-libs
+  - [ ] in firecloud-orchestration
 - [ ] Get two thumbsworth of PR review
 - [ ] Verify all tests go green, including CI tests
 - [ ] **Squash commits and merge** to develop (branches are automatically deleted after merging)

README.md

@@ -79,7 +79,10 @@ After satisfying the above requirements, execute the following command from the
 ./config/docker-rsync-local-rawls.sh
 ```
 
-By default, this will set up an instance of rawls pointing to the database and Sam in dev. 
+**Reminder:** You should be on a Broad internal internet connection or VPN every time you want to run Rawls locally!
+If you are not, Rawls may start up, but certain functionality may not work as expected.
+
+By default, this command will set up an instance of rawls pointing to the database and Sam in dev. 
 It will also set up a process that will watch the local files for changes, and restart the service when source files change.
 
 See docker-rsync-local-rawls.sh for more configuration options.
@@ -99,6 +102,8 @@ BACK_RAWLS=true ./config/docker-rsync-local-rawls.sh
 ```
 
 **Important**: It is highly recommended that use your own Cloud SQL instance when running an instance of back Rawls by cloning the [dev Rawls DB](https://console.cloud.google.com/sql/instances/terraform-qfarbdq3lrexxck5htofjs5z6m/overview?project=broad-dsde-dev). See note below on database work.
+* If you use the dev database itself, your local Rawls instance (with your code changes!) and dev Rawls can interact in
+unpredictable ways, affecting both your own local testing and others' testing in the dev environment.
 
 #### Developing Database Schema Changes
 
@@ -154,19 +159,21 @@ VAULT_TOKEN=$(cat ~/.vault-token) ARTIFACTORY_USERNAME=dsdejenkins ARTIFACTORY_P
 You can view what is in the artifactory here: https://broadinstitute.jfrog.io/broadinstitute/webapp/#/home
 
 After publishing:
-* update [model/CHANGELOG.md](model/CHANGELOG.md) properly
-* update the rawls-model dependency in the automation subdirectory, and ensure that sbt project is healthy
-* update the rawls-model dependency in workbench-libs serviceTest, and ensure that sbt project is healthy
+* Update [model/CHANGELOG.md](model/CHANGELOG.md) properly
+* Update the rawls-model dependency in the automation subdirectory, and ensure that the sbt project is healthy
+* In workbench-libs, update the rawls-model dependency in the project dependencies, update the README and the serviceTest changelog accordingly, and ensure that the sbt project is healthy
+* In firecloud-orchestration, update the rawls-model dependency in the project dependencies, and ensure that the sbt project is healthy
 
 
 ## Troubleshooting
 
 If you get the error message `release version 17 not supported`:
 * Run `java -version` and verify that you're running 17. If not, you will need to install / update your PATH.
 
-If you have trouble submitting workflows and see errors like `HTTP error calling URI https://cromiam-priv.dsde-dev.broadinstitute.org`:
+If you have trouble submitting workflows and see errors like `HTTP error calling URI https://cromiam-priv.dsde-dev.broadinstitute.org`,
+or you get unusual errors for other workflows-related functionality (like accessing workflow configurations):
 * Connect to the NonSplit VPN and try again
-* CromIAM doesn't accept requests from outside the Broad trusted IP space 
+* CromIAM doesn't accept requests from outside the Broad trusted IP space
 
 When running back Rawls with a DB clone, the app may crash on launch with an error related to `OpenTelemetry`.
 * Work around by setting `entityStatisticsCache.enabled = false`.

core/src/main/scala/org/broadinstitute/dsde/rawls/Boot.scala

@@ -299,6 +299,8 @@ object Boot extends IOApp with LazyLogging {
         CromwellBackend(appConfigManager.conf.getString("executionservice.defaultNetworkBackend"))
       val highSecurityNetworkCromwellBackend: CromwellBackend =
         CromwellBackend(appConfigManager.conf.getString("executionservice.highSecurityNetworkBackend"))
+      val gcpBatchBackend: CromwellBackend =
+        CromwellBackend(appConfigManager.conf.getString("executionservice.gcpBatchBackend"))
 
       val wdlParsingConfig = WDLParserConfig(appConfigManager.conf.getConfig("wdl-parsing"))
       def cromwellSwaggerClient = new CromwellSwaggerClient(wdlParsingConfig.serverBasePath)
@@ -599,6 +601,7 @@ object Boot extends IOApp with LazyLogging {
           useWorkflowCollectionLabel,
           defaultNetworkCromwellBackend,
           highSecurityNetworkCromwellBackend,
+          gcpBatchBackend,
           methodConfigResolver,
           bardService,
           workspaceSettingRepository

core/src/main/scala/org/broadinstitute/dsde/rawls/jobexec/WorkflowSubmissionActor.scala

@@ -44,6 +44,7 @@ object WorkflowSubmissionActor {
             useWorkflowCollectionLabel: Boolean,
             defaultNetworkCromwellBackend: CromwellBackend,
             highSecurityNetworkCromwellBackend: CromwellBackend,
+            gcpBatchBackend: CromwellBackend,
             methodConfigResolver: MethodConfigResolver,
             bardService: BardService,
             workspaceSettingRepository: WorkspaceSettingRepository
@@ -69,6 +70,7 @@ object WorkflowSubmissionActor {
         useWorkflowCollectionLabel,
         defaultNetworkCromwellBackend,
         highSecurityNetworkCromwellBackend,
+        gcpBatchBackend,
         methodConfigResolver,
         bardService,
         workspaceSettingRepository
@@ -105,6 +107,7 @@ class WorkflowSubmissionActor(val dataSource: SlickDataSource,
                               val useWorkflowCollectionLabel: Boolean,
                               val defaultNetworkCromwellBackend: CromwellBackend,
                               val highSecurityNetworkCromwellBackend: CromwellBackend,
+                              val gcpBatchBackend: CromwellBackend,
                               val methodConfigResolver: MethodConfigResolver,
                               val bardService: BardService,
                               val workspaceSettingRepository: WorkspaceSettingRepository
@@ -159,6 +162,7 @@ trait WorkflowSubmission extends FutureSupport with LazyLogging with MethodWiths
   val useWorkflowCollectionLabel: Boolean
   val defaultNetworkCromwellBackend: CromwellBackend
   val highSecurityNetworkCromwellBackend: CromwellBackend
+  val gcpBatchBackend: CromwellBackend
   val methodConfigResolver: MethodConfigResolver
   val bardService: BardService
   val workspaceSettingRepository: WorkspaceSettingRepository
@@ -301,7 +305,16 @@ trait WorkflowSubmission extends FutureSupport with LazyLogging with MethodWiths
       // - final_workflow_outputs_dir = submissions/final-outputs
       // - final_workflow_outputs_mode = "copy".
 
+      useCromwellGcpBatchBackend: Boolean = currentSettings
+        .collectFirst { case setting: UseCromwellGcpBatchBackendSetting =>
+          setting
+        }
+        .exists(_.config.enabled)
+      cromwellSubmissionBackend =
+        if (useCromwellGcpBatchBackend) gcpBatchBackend else highSecurityNetworkCromwellBackend
+
       executionServiceWorkflowOptions = ExecutionServiceWorkflowOptions(
+        submission.submissionRoot,
         submission.submissionRoot,
         final_workflow_outputs_dir,
         final_workflow_outputs_dir_metadata,
@@ -315,7 +328,7 @@ trait WorkflowSubmission extends FutureSupport with LazyLogging with MethodWiths
         deleteIntermediateOutputFiles,
         useReferenceDisks,
         memoryRetryMultiplier,
-        highSecurityNetworkCromwellBackend,
+        cromwellSubmissionBackend,
         workflowFailureMode,
         google_labels = Map("terra-submission-id" -> s"terra-${submission.id.toString}"),
         ignoreEmptyOutputs,
@@ -467,6 +480,7 @@ trait WorkflowSubmission extends FutureSupport with LazyLogging with MethodWiths
         monitoringImageScript = submissionRec.monitoringImageScript
       )
     } yield {
+      logger.error("Test workflow options: " + wfOpts.toString);
       val submissionAndWorkspaceLabels =
         Map("submission-id" -> submissionRec.id.toString, "workspace-id" -> workspaceRec.id.toString)
       val wfLabels = workspaceRec.workflowCollection match {

core/src/main/scala/org/broadinstitute/dsde/rawls/monitor/BootMonitors.scala

@@ -94,6 +94,7 @@ object BootMonitors extends LazyLogging {
                    useWorkflowCollectionLabel: Boolean,
                    defaultNetworkCromwellBackend: CromwellBackend,
                    highSecurityNetworkCromwellBackend: CromwellBackend,
+                   gcpBatchBackend: CromwellBackend,
                    methodConfigResolver: MethodConfigResolver,
                    bardService: BardService,
                    workspaceSettingRepository: WorkspaceSettingRepository
@@ -147,6 +148,7 @@ object BootMonitors extends LazyLogging {
         useWorkflowCollectionLabel,
         defaultNetworkCromwellBackend,
         highSecurityNetworkCromwellBackend,
+        gcpBatchBackend,
         methodConfigResolver,
         bardService,
         workspaceSettingRepository
@@ -338,6 +340,7 @@ object BootMonitors extends LazyLogging {
                                             useWorkflowCollectionLabel: Boolean,
                                             defaultNetworkCromwellBackend: CromwellBackend,
                                             highSecurityNetworkCromwellBackend: CromwellBackend,
+                                            gcpBatchBackend: CromwellBackend,
                                             methodConfigResolver: MethodConfigResolver,
                                             bardService: BardService,
                                             workspaceSettingRepository: WorkspaceSettingRepository
@@ -366,6 +369,7 @@ object BootMonitors extends LazyLogging {
           useWorkflowCollectionLabel,
           defaultNetworkCromwellBackend,
           highSecurityNetworkCromwellBackend,
+          gcpBatchBackend,
           methodConfigResolver,
           bardService,
           workspaceSettingRepository

core/src/test/scala/org/broadinstitute/dsde/rawls/jobexec/WorkflowSubmissionSpec.scala

@@ -23,9 +23,11 @@ import org.broadinstitute.dsde.rawls.jobexec.WorkflowSubmissionActor.{
 import org.broadinstitute.dsde.rawls.metrics.{BardService, RawlsStatsDTestUtils}
 import org.broadinstitute.dsde.rawls.mock.{MockBardService, MockSamDAO, RemoteServicesMockServer}
 import org.broadinstitute.dsde.rawls.model.ExecutionJsonSupport.ExecutionServiceWorkflowOptionsFormat
-import org.broadinstitute.dsde.rawls.model.WorkspaceSettingConfig.SeparateSubmissionFinalOutputsConfig
-import org.broadinstitute.dsde.rawls.model.WorkspaceSettingTypes.SeparateSubmissionFinalOutputs
-import org.broadinstitute.dsde.rawls.model.{WorkspaceSetting, _}
+import org.broadinstitute.dsde.rawls.model.WorkspaceSettingConfig.{
+  SeparateSubmissionFinalOutputsConfig,
+  UseCromwellGcpBatchBackendConfig
+}
+import org.broadinstitute.dsde.rawls.model._
 import org.broadinstitute.dsde.rawls.util.MockitoTestUtils
 import org.broadinstitute.dsde.rawls.workspace.WorkspaceSettingRepository
 import org.broadinstitute.dsde.rawls.{RawlsExceptionWithErrorReport, RawlsTestUtils}
@@ -120,6 +122,7 @@ class WorkflowSubmissionSpec(_system: ActorSystem)
     val useWorkflowCollectionLabel: Boolean = false,
     val defaultNetworkCromwellBackend: CromwellBackend = CromwellBackend("PAPIv2"),
     val highSecurityNetworkCromwellBackend: CromwellBackend = CromwellBackend("PAPIv2-CloudNAT"),
+    val gcpBatchBackend: CromwellBackend = CromwellBackend("GCPBatch"),
     val methodConfigResolver: MethodConfigResolver = methodConfigResolver,
     val bardService: BardService = mockBardService,
     val workspaceSettingRepository: WorkspaceSettingRepository = mockWorkspaceSettingRepository
@@ -381,6 +384,7 @@ class WorkflowSubmissionSpec(_system: ActorSystem)
         Some(
           ExecutionServiceWorkflowOptions(
             jes_gcs_root = s"gs://${testData.workspace.bucketName}/${testData.submission1.submissionId}",
+            gcp_batch_gcs_root = s"gs://${testData.workspace.bucketName}/${testData.submission1.submissionId}",
             None,
             None,
             google_project = testData.workspace.googleProjectId.value,
@@ -937,6 +941,7 @@ class WorkflowSubmissionSpec(_system: ActorSystem)
           false,
           CromwellBackend("PAPIv2"),
           CromwellBackend("PAPIv2-CloudNAT"),
+          CromwellBackend("GCPBatch"),
           methodConfigResolver,
           mockBardService,
           mockWorkspaceSettingRepository
@@ -1005,6 +1010,7 @@ class WorkflowSubmissionSpec(_system: ActorSystem)
           false,
           CromwellBackend("PAPIv2"),
           CromwellBackend("PAPIv2-CloudNAT"),
+          CromwellBackend("GCPBatch"),
           methodConfigResolver,
           mockBardService,
           mockWorkspaceSettingRepository
@@ -1202,6 +1208,90 @@ class WorkflowSubmissionSpec(_system: ActorSystem)
     }
   }
 
+  it should "submit workflows to Cromwell's GCP Batch backend when UseCromwellGcpBatchBackendSetting is true" in withDefaultTestDatabase {
+    val mockExecCluster = MockShardedExecutionServiceCluster.fromDAO(new MockExecutionServiceDAO(), slickDataSource)
+    val workspaceSettingRepository = mock[WorkspaceSettingRepository]
+    when(workspaceSettingRepository.getWorkspaceSettings(UUID.fromString(testData.workspace.workspaceId))).thenReturn(
+      Future.successful(List(UseCromwellGcpBatchBackendSetting(UseCromwellGcpBatchBackendConfig(true))))
+    )
+
+    val workflowSubmission =
+      new TestWorkflowSubmission(slickDataSource, workspaceSettingRepository = workspaceSettingRepository) {
+        override val executionServiceCluster = mockExecCluster
+      }
+
+    val (workflowRecs, submissionRec, workspaceRec) =
+      getWorkflowSubmissionWorkspaceRecords(testData.regionalSubmission, testData.workspace)
+
+    Await.result(
+      workflowSubmission.submitWorkflowBatch(WorkflowBatch(workflowRecs.map(_.id), submissionRec, workspaceRec)),
+      Duration.Inf
+    )
+
+    val workflowOptions = mockExecCluster.getDefaultSubmitMember
+      .asInstanceOf[MockExecutionServiceDAO]
+      .submitOptions
+      .map(_.parseJson.convertTo[ExecutionServiceWorkflowOptions])
+
+    workflowOptions.get.backend should be(CromwellBackend("GCPBatch"))
+  }
+
+  it should "submit workflows to Cromwell's high security network backend when UseCromwellGcpBatchBackendSetting is false" in withDefaultTestDatabase {
+    val mockExecCluster = MockShardedExecutionServiceCluster.fromDAO(new MockExecutionServiceDAO(), slickDataSource)
+    val workspaceSettingRepository = mock[WorkspaceSettingRepository]
+    when(workspaceSettingRepository.getWorkspaceSettings(UUID.fromString(testData.workspace.workspaceId))).thenReturn(
+      Future.successful(List(UseCromwellGcpBatchBackendSetting(UseCromwellGcpBatchBackendConfig(false))))
+    )
+
+    val workflowSubmission =
+      new TestWorkflowSubmission(slickDataSource, workspaceSettingRepository = workspaceSettingRepository) {
+        override val executionServiceCluster = mockExecCluster
+      }
+
+    val (workflowRecs, submissionRec, workspaceRec) =
+      getWorkflowSubmissionWorkspaceRecords(testData.regionalSubmission, testData.workspace)
+
+    Await.result(
+      workflowSubmission.submitWorkflowBatch(WorkflowBatch(workflowRecs.map(_.id), submissionRec, workspaceRec)),
+      Duration.Inf
+    )
+
+    val workflowOptions = mockExecCluster.getDefaultSubmitMember
+      .asInstanceOf[MockExecutionServiceDAO]
+      .submitOptions
+      .map(_.parseJson.convertTo[ExecutionServiceWorkflowOptions])
+
+    workflowOptions.get.backend should be(CromwellBackend("PAPIv2-CloudNAT"))
+  }
+
+  it should "submit workflows to Cromwell's high security network backend when UseCromwellGcpBatchBackendSetting is not set" in withDefaultTestDatabase {
+    val mockExecCluster = MockShardedExecutionServiceCluster.fromDAO(new MockExecutionServiceDAO(), slickDataSource)
+    val workspaceSettingRepository = mock[WorkspaceSettingRepository]
+    when(workspaceSettingRepository.getWorkspaceSettings(UUID.fromString(testData.workspace.workspaceId))).thenReturn(
+      Future.successful(List())
+    )
+
+    val workflowSubmission =
+      new TestWorkflowSubmission(slickDataSource, workspaceSettingRepository = workspaceSettingRepository) {
+        override val executionServiceCluster = mockExecCluster
+      }
+
+    val (workflowRecs, submissionRec, workspaceRec) =
+      getWorkflowSubmissionWorkspaceRecords(testData.regionalSubmission, testData.workspace)
+
+    Await.result(
+      workflowSubmission.submitWorkflowBatch(WorkflowBatch(workflowRecs.map(_.id), submissionRec, workspaceRec)),
+      Duration.Inf
+    )
+
+    val workflowOptions = mockExecCluster.getDefaultSubmitMember
+      .asInstanceOf[MockExecutionServiceDAO]
+      .submitOptions
+      .map(_.parseJson.convertTo[ExecutionServiceWorkflowOptions])
+
+    workflowOptions.get.backend should be(CromwellBackend("PAPIv2-CloudNAT"))
+  }
+
   private def setWorkflowBatchToQueued(batchSize: Int, submissionId: String): Seq[WorkflowRecord] =
     runAndWait(
       for {

core/src/test/scala/org/broadinstitute/dsde/rawls/model/ExecutionModelSpec.scala

@@ -126,6 +126,7 @@ class ExecutionModelSpec extends AnyFlatSpec with Matchers {
   "ExecutionServiceWorkflowOptions" should "serialize/deserialize to/from JSON" in {
     val test = ExecutionServiceWorkflowOptions(
       jes_gcs_root = "jes_gcs_root",
+      gcp_batch_gcs_root = "example_gcp_batch_gcs_root",
       final_workflow_outputs_dir = None,
       final_workflow_outputs_dir_metadata = None,
       google_project = "google_project",
@@ -162,6 +163,7 @@ class ExecutionModelSpec extends AnyFlatSpec with Matchers {
       """
         |{
         |  "jes_gcs_root": "jes_gcs_root",
+        |  "gcp_batch_gcs_root": "example_gcp_batch_gcs_root",
         |  "google_project": "google_project",
         |  "account_name": "account_name",
         |  "google_compute_service_account": "[email protected]",
@@ -192,6 +194,7 @@ class ExecutionModelSpec extends AnyFlatSpec with Matchers {
       """
         |{
         |  "jes_gcs_root": "jes_gcs_root",
+        |  "gcp_batch_gcs_root": "example_gcp_batch_gcs_root",
         |  "google_project": "google_project",
         |  "account_name": "account_name",
         |  "google_compute_service_account": "[email protected]",

core/src/test/scala/org/broadinstitute/dsde/rawls/webservice/SubmissionApiServiceSpec.scala

@@ -115,6 +115,7 @@ class SubmissionApiServiceSpec extends ApiServiceSpec with TableDrivenPropertyCh
         false,
         CromwellBackend("PAPIv2"),
         CromwellBackend("PAPIv2-CloudNAT"),
+        CromwellBackend("GCPBatch"),
         methodConfigResolver,
         new MockBardService(),
         new WorkspaceSettingRepository(slickDataSource)

local-dev/templates/rawls.conf

@@ -93,8 +93,8 @@ dataRepo {
 
 resourceBuffer {
   projectPool {
-    regular = "cwb_ws_dev_v7"
-    exfiltrationControlled = "vpc_sc_v10"
+    regular = "cwb_ws_dev_v8"
+    exfiltrationControlled = "vpc_sc_v13"
   }
   url = "https://buffer.dsde-dev.broadinstitute.org"
   saEmail = "[email protected]"
@@ -132,6 +132,7 @@ executionservice {
 
   defaultNetworkBackend = "PAPIv2-beta"
   highSecurityNetworkBackend = "PAPIv2-CloudNAT"
+  gcpBatchBackend = "GCPBatch"
 
   cromiamUrl = "https://cromiam-priv.dsde-dev.broadinstitute.org"
 }