Terraform for GCP CI base #144

AlCutter · 2024-08-15T17:14:46Z

This PR adds support for spinning up an instance of the example personality on GCP.

This is intended to be used for CI tests.

Towards #7

codecov-commenter · 2024-08-15T17:20:24Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 35.28%. Comparing base (46ec9c2) to head (9c1ec4b).
Report is 72 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #144      +/-   ##
==========================================
- Coverage   35.80%   35.28%   -0.53%     
==========================================
  Files          16       33      +17     
  Lines        1363     2874    +1511     
==========================================
+ Hits          488     1014     +526     
- Misses        801     1750     +949     
- Partials       74      110      +36

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

mhutchinson · 2024-08-20T16:06:35Z

deployment/live/gcp/example-gcp/ci/terragrunt.hcl

@@ -0,0 +1,16 @@
+terraform {
+  source = "${get_repo_root()}/deployment/modules/gcp//example-gcp"


This double slash is a bit weird?

This is how you tell terraform about module "boundaries", with the double slash it brings in anything under there, without them it fails when you try to reference the sibling gcs from within the example-gc module.

This is a bit of a dusty corner in terraform, but it's why e.g. you had to put double slashes in for your distributor config which referenced other modules: https://github.com/transparency-dev/distributor/blob/main/deployment/modules/distributor/main.tf#L110

mhutchinson · 2024-08-20T16:09:27Z

deployment/live/gcp/example-gcp/terragrunt.hcl

  project_id = get_env("GOOGLE_PROJECT", "trillian-tessera")
  location   = get_env("GOOGLE_REGION", "us-central1")
-  base_name   = get_env("TESSERA_BASE_NAME", "example-gcp")
+  base_name   = get_env("TESSERA_BASE_NAME", "${path_relative_to_include()}-example-gcp")


Can you use $env here instead of the path calculation? It'd be clearer for a human reading it, even if the result is the same.

I assumed I couldn't, but rather unexpectedly it seems like you can (or at least ${local.env} you can)...

mhutchinson · 2024-08-20T16:09:57Z

deployment/live/gcp/example-gcp/terragrunt.hcl

 }

 locals {
+  env         = path_relative_to_include()
  project_id = get_env("GOOGLE_PROJECT", "trillian-tessera")


nit: alignment of = has got wonky.

mhutchinson · 2024-08-20T16:10:39Z

deployment/live/gcp/example-gcp/terragrunt.hcl

 remote_state {
  backend = "gcs"

  config = {
    project  = local.project_id
    location = local.location
    bucket   = "${local.project_id}-${local.base_name}-terraform-state"
+    prefix   = "${path_relative_to_include()}/terraform.tfstate"


Again, using ${local.env} would be clearer, IMO.

mhutchinson · 2024-08-20T16:14:15Z

deployment/modules/gcp/example-gcp/main.tf

+##
+## This will configure all the storage infrastructure required to run a Tessera log on GCP.
+module "gcs" {
+  source = "..//gcs"


Is the double slash needed here? I think this is one of those cases where there is something needed about the 2 slashes but best to check.

yup, same as above.

mhutchinson · 2024-08-20T16:17:59Z

deployment/modules/gcp/example-gcp/main.tf

+    containers {
+      image      = "us-docker.pkg.dev/cloud-ops-agents-artifacts/cloud-run-gmp-sidecar/cloud-run-gmp-sidecar:1.0.0"
+      name       = "collector"
+      depends_on = ["example-gcp"]
+    }


This does the prometheus integration into GCP monitoring. Not sure if you need it in the example?

It could do with some comments put through here for things like this though, or people will just steal all of this whether it's needed or not. The curse of writing examples that try to strike a balance between "hello world" and showing actual real-life deployment.

mhutchinson · 2024-08-20T16:19:19Z

deployment/modules/gcp/example-gcp/main.tf

+        "--logtostderr",
+        "--v=1",
+        "--bucket=${module.gcs.log_bucket.id}",
+        "--spanner=projects/${var.project_id}/instances/${module.gcs.log_spanner_instance.name}/databases/${module.gcs.log_spanner_db.name}",


nit: You could consider evaluation this monster as a local, and even exporting it as an output variable. It would make this line shorter and thus easier to eyeball the flags.

mhutchinson · 2024-08-20T16:21:45Z

deployment/modules/gcp/example-gcp/main.tf

+resource "google_service_account" "cloudrun_service_account" {
+  account_id   = "cloudrun-${var.env}-sa"
+  display_name = "Service Account for Cloud Run (${var.env})"
+}


This isn't strictly necessary as there's a default service account you could use. It's good practice though. We should probably write in some docs our approach to the example on how to balance best practice vs minimalism.

Yeah, guidance is a good idea. I'll merge this as-is while we chew on that.

AlCutter force-pushed the gcp_tf_ci_base branch 29 times, most recently from b94d163 to cc921d2 Compare August 19, 2024 13:57

AlCutter force-pushed the gcp_tf_ci_base branch 7 times, most recently from 5e05e32 to 179a1fe Compare August 19, 2024 15:47

AlCutter changed the title ~~WIP: GCP CI base~~ Terraform for GCP CI base Aug 19, 2024

AlCutter force-pushed the gcp_tf_ci_base branch from fb0b06e to fffe57b Compare August 20, 2024 12:07

AlCutter mentioned this pull request Aug 20, 2024

Add support for using GCP KMS in example-gcp #153

Merged

AlCutter force-pushed the gcp_tf_ci_base branch 4 times, most recently from db5632a to b20a73e Compare August 20, 2024 15:28

AlCutter added 3 commits August 20, 2024 16:34

Add example-gcp module

8767fb6

Docker image name

8e91197

Add support for KMS

bba2f74

AlCutter force-pushed the gcp_tf_ci_base branch from b20a73e to bba2f74 Compare August 20, 2024 15:34

Update README

254df25

AlCutter marked this pull request as ready for review August 20, 2024 15:37

AlCutter requested a review from mhutchinson August 20, 2024 15:37

AlCutter assigned mhutchinson Aug 20, 2024

mhutchinson approved these changes Aug 20, 2024

View reviewed changes

AlCutter force-pushed the gcp_tf_ci_base branch from 71a1ebe to 51f0e90 Compare August 20, 2024 16:40

Try referencing env

ab8c657

AlCutter force-pushed the gcp_tf_ci_base branch from 51f0e90 to ab8c657 Compare August 20, 2024 16:41

fixes

9c1ec4b

AlCutter force-pushed the gcp_tf_ci_base branch from 3db7706 to 9c1ec4b Compare August 20, 2024 16:51

AlCutter merged commit 38226b6 into transparency-dev:main Aug 20, 2024
8 checks passed

AlCutter deleted the gcp_tf_ci_base branch August 20, 2024 16:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Terraform for GCP CI base #144

Terraform for GCP CI base #144

AlCutter commented Aug 15, 2024 •

edited

Loading

codecov-commenter commented Aug 15, 2024 •

edited

Loading

mhutchinson Aug 20, 2024

AlCutter Aug 20, 2024 •

edited

Loading

mhutchinson Aug 20, 2024

AlCutter Aug 20, 2024

mhutchinson Aug 20, 2024

mhutchinson Aug 20, 2024

AlCutter Aug 20, 2024

mhutchinson Aug 20, 2024

AlCutter Aug 20, 2024

mhutchinson Aug 20, 2024

mhutchinson Aug 20, 2024

mhutchinson Aug 20, 2024

AlCutter Aug 20, 2024

		@@ -0,0 +1,16 @@
		terraform {
		source = "${get_repo_root()}/deployment/modules/gcp//example-gcp"

Terraform for GCP CI base #144

Terraform for GCP CI base #144

Conversation

AlCutter commented Aug 15, 2024 • edited Loading

codecov-commenter commented Aug 15, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

AlCutter Aug 20, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AlCutter commented Aug 15, 2024 •

edited

Loading

codecov-commenter commented Aug 15, 2024 •

edited

Loading

AlCutter Aug 20, 2024 •

edited

Loading