feat!: migrating to TheLook Ecommerce dataset #257

shanecglass · 2023-10-02T15:13:32Z

No description provided.

Initial migration to thelook

davenportjw

good start! a few things around dependencies and what the linter will dislike to resolve.

davenportjw · 2023-10-05T14:28:44Z

modules/data_warehouse/bigquery.tf

+
+  labels = var.labels
+
+  depends_on = [


we can remove these since the solution calls out the resources specifically above (applies to other tables below).

Cleaned up these dependencies, as well as those across all .tf files

davenportjw · 2023-10-05T14:29:23Z

modules/data_warehouse/bigquery.tf

+  table_id            = "events"
+  project             = module.project-services.project_id
+  deletion_protection = var.deletion_protection
+  # max_staleness =  "1:0:0"


if we arent planning to use these, I'd probably just remove them to keep the code simple.

I kept them in here because I thought it would be good for folks to see the syntax, but commented them out so that the Looker Studio report works immediately.

modules/data_warehouse/bigquery.tf

davenportjw · 2023-10-05T14:30:56Z

modules/data_warehouse/bigquery.tf

@@ -182,6 +305,8 @@ resource "google_project_iam_member" "dts_roles" {
  project = module.project-services.project_id
  role    = each.key
  member  = "serviceAccount:${google_service_account.dts.email}"
+
+  depends_on = [google_service_account.dts]


this one isnt needed since you have an output attribute of DTS resource on 307

davenportjw · 2023-10-05T14:31:43Z

modules/data_warehouse/src/sql/sp_bigqueryml_model.sql

+    )
+    GROUP BY user_id
+  )
+);


add new lines

davenportjw · 2023-10-05T14:32:25Z

modules/data_warehouse/variables.tf

@@ -39,13 +39,13 @@ variable "enable_apis" {
 variable "force_destroy" {
  type        = string
  description = "Whether or not to protect BigQuery resources from deletion when solution is modified or changed."
-  default     = false
+  default     = true


we set these to the non-destructive option by default in the code (we override them in Solution Deployment)

davenportjw · 2023-10-05T14:32:31Z

modules/data_warehouse/variables.tf

 }

 variable "deletion_protection" {
  type        = string
  description = "Whether or not to protect GCS resources from deletion when solution is modified or changed."
-  default     = true
+  default     = false


same as above.

davenportjw · 2023-10-05T14:32:43Z

modules/data_warehouse/workflows.tf

  project = module.project-services.project_id
  role    = each.key
  member  = "serviceAccount:${google_service_account.workflow_service_account.email}"

+
+  depends_on = [google_service_account.workflow_service_account]


not needed.

davenportjw · 2023-10-05T15:05:29Z

Gotcha. I'd probably remove them to just keep the code simpler and we can look at if we need to add an example in the terraform resource provider.

…

On Thu, Oct 5, 2023 at 8:58 AM Shane Glass ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In modules/data_warehouse/bigquery.tf <#257 (comment)> : > @@ -48,22 +51,127 @@ resource "google_storage_bucket_iam_binding" "bq_connection_iam_object_viewer" { ] } -# # Create a BigQuery external table -resource "google_bigquery_table" "tbl_edw_taxi" { +# # Create a Biglake table for events with metadata caching +resource "google_bigquery_table" "tbl_edw_events" { + dataset_id = google_bigquery_dataset.ds_edw.dataset_id + table_id = "events" + project = module.project-services.project_id + deletion_protection = var.deletion_protection + # max_staleness = "1:0:0" I kept them in here because I thought it would be good for folks to see the syntax, but commented them out so that the Looker Studio report works immediately. — Reply to this email directly, view it on GitHub <#257 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADK7DPSPA3PZTM4ZSJ3NQWLX53DHXAVCNFSM6AAAAAA5PSUEBOVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMYTMNRQGA4TINZTGI> . You are receiving this because your review was requested.Message ID: <terraform-google-modules/terraform-google-bigquery/pull/257/review/1660094732 @github.com>

davenportjw · 2023-10-05T15:07:05Z

/gcbrun

davenportjw · 2023-10-09T14:19:29Z

/gcbrun

davenportjw · 2023-10-09T14:23:18Z

/gcbrun

…dules-master' into terraform-google-modules-master

davenportjw · 2023-10-09T17:14:03Z

/gcbrun

davenportjw · 2023-10-09T18:48:15Z

/gcbrun

davenportjw · 2023-10-09T19:05:34Z

/gcbrun

commit f88f4b5 Author: Shane Glass <[email protected]> Date: Mon Oct 23 15:10:22 2023 -0400 feat: data_warehouse Add GenAI capabilities (terraform-google-modules#272) commit b7efc4d Author: Awais Malik <[email protected]> Date: Thu Oct 19 15:38:43 2023 -0700 fix: adds a null check for expiration time (terraform-google-modules#268) commit 405972a Author: Shane Glass <[email protected]> Date: Tue Oct 17 17:03:10 2023 -0400 fix: update workflow.tftpl (terraform-google-modules#266) commit 1046e0d Author: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Date: Tue Oct 17 09:29:48 2023 -0700 chore(deps): update cft/developer-tools docker tag to v1.17 (terraform-google-modules#265) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> commit ceef798 Author: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com> Date: Tue Oct 17 09:48:20 2023 -0600 chore(master): release 7.0.0 (terraform-google-modules#249) commit 1a7620b Author: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Date: Tue Oct 10 09:11:32 2023 -0700 chore(deps): Update module github.com/GoogleCloudPlatform/cloud-foundation-toolkit/infra/blueprint-test to v0.9.0 (terraform-google-modules#260) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> commit 7fd5bcb Author: Awais Malik <[email protected]> Date: Mon Oct 9 14:33:52 2023 -0700 fix: upgraded versions.tf to include minor bumps from tpg v5 (terraform-google-modules#261) Co-authored-by: Andrew Peabody <[email protected]> Co-authored-by: Jason Davenport <[email protected]> commit e97adfb Author: Shane Glass <[email protected]> Date: Mon Oct 9 15:27:46 2023 -0400 feat!: data_warehosue migrating to TheLook Ecommerce dataset (terraform-google-modules#257) commit 096ca4e Author: Lloyd Armstrong <[email protected]> Date: Mon Oct 9 18:28:35 2023 +0200 fix: upgrade hashicorp/google to 5.0 (terraform-google-modules#259) Co-authored-by: Lloyd Armstrong <[email protected]>

Squashed commit of the following: commit f88f4b5 Author: Shane Glass <[email protected]> Date: Mon Oct 23 15:10:22 2023 -0400 feat: data_warehouse Add GenAI capabilities (terraform-google-modules#272) commit b7efc4d Author: Awais Malik <[email protected]> Date: Thu Oct 19 15:38:43 2023 -0700 fix: adds a null check for expiration time (terraform-google-modules#268) commit 405972a Author: Shane Glass <[email protected]> Date: Tue Oct 17 17:03:10 2023 -0400 fix: update workflow.tftpl (terraform-google-modules#266) commit 1046e0d Author: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Date: Tue Oct 17 09:29:48 2023 -0700 chore(deps): update cft/developer-tools docker tag to v1.17 (terraform-google-modules#265) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> commit ceef798 Author: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com> Date: Tue Oct 17 09:48:20 2023 -0600 chore(master): release 7.0.0 (terraform-google-modules#249) commit 1a7620b Author: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Date: Tue Oct 10 09:11:32 2023 -0700 chore(deps): Update module github.com/GoogleCloudPlatform/cloud-foundation-toolkit/infra/blueprint-test to v0.9.0 (terraform-google-modules#260) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> commit 7fd5bcb Author: Awais Malik <[email protected]> Date: Mon Oct 9 14:33:52 2023 -0700 fix: upgraded versions.tf to include minor bumps from tpg v5 (terraform-google-modules#261) Co-authored-by: Andrew Peabody <[email protected]> Co-authored-by: Jason Davenport <[email protected]> commit e97adfb Author: Shane Glass <[email protected]> Date: Mon Oct 9 15:27:46 2023 -0400 feat!: data_warehosue migrating to TheLook Ecommerce dataset (terraform-google-modules#257) commit 096ca4e Author: Lloyd Armstrong <[email protected]> Date: Mon Oct 9 18:28:35 2023 +0200 fix: upgrade hashicorp/google to 5.0 (terraform-google-modules#259) Co-authored-by: Lloyd Armstrong <[email protected]>

…rm-google-modules#257)

shanecglass added 30 commits August 21, 2023 16:29

Initial changes

eb68dcb

initial changes

dd0e158

Merge pull request #1 from shanecglass/scg-dev

5831fa2

Initial migration to thelook

Add RANK() to sample query

22271ea

Add waiter after APIs and clarify dependencies

dd4ac1d

Convert tables to Biglake tables

d27e903

Updating outputs

d8a45b4

Fixing bug

9a559f8

Debug

cd6ad43

Typo

4b452cd

Adding max staleness refresh

359148c

correcting errors

9f77d83

Testing

1eb414f

testing

2277866

testing bug fix

0fddd9d

bug testing

2e94bb0

bug testing

e0e76da

testing max staleness

0fc4a22

staleness testing

7b63f3f

staleness testing

d7bc72d

staleness bug testing

ed0dd4a

staleness bug testing

8ffee33

updating terraform version required

559b426

staleness bug testing

7353572

staleness bug testing

afb5483

Correcting caching interval error

7e553ad

Cleanup sample queries

b3a8516

Clean up sample queries

bf63e23

Correcting typo

db37112

Update Looker Studio template link

051c1f2

shanecglass requested a review from a team as a code owner October 2, 2023 15:13

shanecglass changed the title ~~Merging deconflicted changes~~ feat!: migrating to TheLook Ecommerce dataset Oct 2, 2023

davenportjw requested changes Oct 5, 2023

View reviewed changes

shanecglass and others added 4 commits October 5, 2023 14:31

Changes for review and lint

1f03801

Merging in changes for review and lint corrections

7180b9b

Correcting count_of_orders calculation

747ffa9

Merge branch 'master' into terraform-google-modules-master

37aa014

shanecglass and others added 4 commits October 9, 2023 12:57

Change BigQuery dataset name

3faa38f

Update renovate.json

9b40a01

Merge remote-tracking branch 'refs/remotes/origin/terraform-google-mo…

080b796

…dules-master' into terraform-google-modules-master

Merge branch 'master' into terraform-google-modules-master

ec2527d

shanecglass added 3 commits October 9, 2023 14:16

Standardize parameters for dataset & project id

fb52bc6

Correcting parameter error

781c1e4

Parameterize dataset_id in workflow

9e7aa8b

davenportjw approved these changes Oct 9, 2023

View reviewed changes

davenportjw merged commit e97adfb into terraform-google-modules:master Oct 9, 2023
4 checks passed

release-please bot mentioned this pull request Oct 9, 2023

chore(master): release 7.0.0 #249

Merged

shanecglass deleted the terraform-google-modules-master branch October 9, 2023 20:02

shanecglass restored the terraform-google-modules-master branch October 9, 2023 20:02

shanecglass deleted the terraform-google-modules-master branch October 9, 2023 20:03

msgongora pushed a commit to cbsi-dto/tf-mod-bq that referenced this pull request Oct 30, 2023

feat!: data_warehosue migrating to TheLook Ecommerce dataset (terrafo…

d06fd0a

…rm-google-modules#257)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat!: migrating to TheLook Ecommerce dataset #257

feat!: migrating to TheLook Ecommerce dataset #257

shanecglass commented Oct 2, 2023

davenportjw left a comment

davenportjw Oct 5, 2023

shanecglass Oct 5, 2023

davenportjw Oct 5, 2023

shanecglass Oct 5, 2023

davenportjw Oct 5, 2023

davenportjw Oct 5, 2023

davenportjw Oct 5, 2023

davenportjw Oct 5, 2023

davenportjw Oct 5, 2023

davenportjw commented Oct 5, 2023 via email

davenportjw commented Oct 5, 2023

davenportjw commented Oct 9, 2023

davenportjw commented Oct 9, 2023

davenportjw commented Oct 9, 2023

davenportjw commented Oct 9, 2023

davenportjw commented Oct 9, 2023

feat!: migrating to TheLook Ecommerce dataset #257

feat!: migrating to TheLook Ecommerce dataset #257

Conversation

shanecglass commented Oct 2, 2023

davenportjw left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davenportjw commented Oct 5, 2023 via email

davenportjw commented Oct 5, 2023

davenportjw commented Oct 9, 2023

davenportjw commented Oct 9, 2023

davenportjw commented Oct 9, 2023

davenportjw commented Oct 9, 2023

davenportjw commented Oct 9, 2023