Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

12/4/2024 Production Deploy - large batch hotfixes #1460

Merged
merged 9 commits into from
Dec 4, 2024
Merged

Conversation

ccostino
Copy link
Contributor

@ccostino ccostino commented Dec 4, 2024

A note to PR reviewers: it may be helpful to review our code review documentation to know what to keep in mind while reviewing pull requests.

Description

This is a hotfix to address a couple of critical issues with our production system that we encountered after the two large batch sends yesterday. The critical piece is adjusting the database inserts to only happen once.

Security Considerations

  • These hotfixes are in response to large volume workloads that are now flowing through our system and will improve application stability.

Kenneth Kehl and others added 7 commits December 3, 2024 12:44
This changeset updates our restage workflow and GitHub action to use the latest version of the cg-cli-tools to help prevent future issues with performing restage actions for our apps.

Signed-off-by: Carlo Costino <[email protected]>
change create notification to only insert once
This changeset boosts our production Redis configuration to the 5node-large plan to give us a bit more wiggle room for heavy production workloads.  This should still keep us in the threshold of the second block of 10 nodes we are already signed up for as we currently have 12 nodes going across all environments at the moment.

Signed-off-by: Carlo Costino <[email protected]>
@ccostino ccostino added engineering python Pull requests that update Python code labels Dec 4, 2024
@ccostino ccostino self-assigned this Dec 4, 2024
stvnrlly
stvnrlly previously approved these changes Dec 4, 2024
Update restage workflow to use latest cg-cli-tools
Bump production Redis plan to 5node-large
Copy link

github-actions bot commented Dec 4, 2024

Terraform Format and Style: success
Terraform Initialization: success
Terraform Validation: success
Terraform Plan: success

Show Plan

null_resource.prevent_destroy: Refreshing state... [id=8795387329123913937]
module.database.data.cloudfoundry_service.rds: Reading...
module.sns_sms.data.cloudfoundry_service.sns: Reading...
module.database.data.cloudfoundry_space.space: Reading...
module.ses_email.data.cloudfoundry_space.space: Reading...
module.egress-space.data.cloudfoundry_org.org: Reading...
module.csv_upload_bucket.data.cloudfoundry_space.space: Reading...
module.redis-v70.data.cloudfoundry_service.redis: Reading...
module.csv_upload_bucket.data.cloudfoundry_service.s3: Reading...
module.redis-v70.data.cloudfoundry_space.space: Reading...
module.sns_sms.data.cloudfoundry_space.space: Reading...
module.sns_sms.data.cloudfoundry_service.sns: Read complete after 0s [id=37cbe836-d604-45d1-b1f8-1403f7cb3ff7]
module.ses_email.data.cloudfoundry_service.ses: Reading...
module.database.data.cloudfoundry_service.rds: Read complete after 0s [id=dcfb1d43-f22c-42d3-962c-7ae04eda24e7]
module.csv_upload_bucket.data.cloudfoundry_service.s3: Read complete after 0s [id=440ce9d9-b108-4bbe-80b4-08338f3cc25b]
module.redis-v70.data.cloudfoundry_service.redis: Read complete after 0s [id=045871d3-d9a8-4a9b-a844-00a067fab054]
module.ses_email.data.cloudfoundry_service.ses: Read complete after 0s [id=8bf5287d-587b-4696-8e53-a6ffdba211c0]
module.egress-space.data.cloudfoundry_org.org: Read complete after 0s [id=9e428562-a2d9-41b4-9c23-1ef5237fb44e]
module.egress-space.data.cloudfoundry_user.users["dd4002f6-951d-44b2-b88c-1b828e98dc01"]: Reading...
module.egress-space.cloudfoundry_space.public_egress: Refreshing state... [id=63506464-c7e2-4743-a7b3-4a8d45f14fb9]
module.sns_sms.data.cloudfoundry_space.space: Read complete after 0s [id=4447d89a-9735-4d82-94e1-dc07713aa460]
module.sns_sms.cloudfoundry_service_instance.sns: Refreshing state... [id=ccaf756d-74a2-4903-b9cf-08ad0123ecbc]
module.database.data.cloudfoundry_space.space: Read complete after 0s [id=4447d89a-9735-4d82-94e1-dc07713aa460]
module.database.cloudfoundry_service_instance.rds: Refreshing state... [id=6267206c-b5ab-4dbb-8662-02f2ada85876]
module.redis-v70.data.cloudfoundry_space.space: Read complete after 0s [id=4447d89a-9735-4d82-94e1-dc07713aa460]
module.redis-v70.cloudfoundry_service_instance.redis: Refreshing state... [id=dd523fa2-6bca-49ac-91b2-b81d77edc787]
module.csv_upload_bucket.data.cloudfoundry_space.space: Read complete after 0s [id=4447d89a-9735-4d82-94e1-dc07713aa460]
module.csv_upload_bucket.cloudfoundry_service_instance.bucket: Refreshing state... [id=0d3aa4e5-7e26-4eef-bf6a-f12818b8a687]
module.ses_email.data.cloudfoundry_space.space: Read complete after 0s [id=4447d89a-9735-4d82-94e1-dc07713aa460]
module.ses_email.cloudfoundry_service_instance.ses: Refreshing state... [id=36cf8e1d-f2b7-4b44-9ef6-998393cb547f]
module.egress-space.data.cloudfoundry_user.users["dd4002f6-951d-44b2-b88c-1b828e98dc01"]: Read complete after 0s [id=c51e169a-d8d0-4179-adff-ef871ba9e8ae]
module.egress-space.cloudfoundry_space_users.deployers: Refreshing state... [id=11c85b1d-3895-b7a0-260c-a036d272226d]

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # module.redis-v70.cloudfoundry_service_instance.redis will be updated in-place
  ~ resource "cloudfoundry_service_instance" "redis" {
        id                             = "dd523fa2-6bca-49ac-91b2-b81d77edc787"
        name                           = "notify-api-redis-v70-production"
      ~ service_plan                   = "a5800f92-4fef-42f9-beb8-afac20fae841" -> "78209316-08fd-451d-bc2c-7d10171b6828"
        tags                           = []
        # (5 unchanged attributes hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

─────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't
guarantee to take exactly these actions if you run "terraform apply" now.

Pusher: @stvnrlly, Action: pull_request

Copy link

github-actions bot commented Dec 4, 2024

Terraform Format and Style: success
Terraform Initialization: success
Terraform Validation: success
Terraform Plan: success

Show Plan

null_resource.prevent_destroy: Refreshing state... [id=6074818404547836262]
module.egress-space.data.cloudfoundry_org.org: Reading...
module.sns_sms.data.cloudfoundry_space.space: Reading...
module.ses_email.data.cloudfoundry_space.space: Reading...
module.database.data.cloudfoundry_space.space: Reading...
module.ses_email.data.cloudfoundry_service.ses: Reading...
module.redis-v70.data.cloudfoundry_space.space: Reading...
module.csv_upload_bucket.data.cloudfoundry_service.s3: Reading...
module.redis-v70.data.cloudfoundry_service.redis: Reading...
module.database.data.cloudfoundry_service.rds: Reading...
module.sns_sms.data.cloudfoundry_service.sns: Reading...
module.sns_sms.data.cloudfoundry_service.sns: Read complete after 1s [id=ed5b0e74-dacc-4511-8514-410bef51b554]
module.csv_upload_bucket.data.cloudfoundry_space.space: Reading...
module.csv_upload_bucket.data.cloudfoundry_service.s3: Read complete after 1s [id=440ce9d9-b108-4bbe-80b4-08338f3cc25b]
module.ses_email.data.cloudfoundry_service.ses: Read complete after 1s [id=8a75c59c-0c4b-4764-907c-b0760c7278a0]
module.egress-space.data.cloudfoundry_org.org: Read complete after 1s [id=9e428562-a2d9-41b4-9c23-1ef5237fb44e]
module.egress-space.data.cloudfoundry_user.users["[email protected]"]: Reading...
module.egress-space.data.cloudfoundry_user.users["bbb35677-6906-4f8d-b230-bb4755d7886a"]: Reading...
module.egress-space.cloudfoundry_space.public_egress: Refreshing state... [id=fa2417a9-297e-40e0-a517-f4f1f4989556]
module.redis-v70.data.cloudfoundry_service.redis: Read complete after 1s [id=045871d3-d9a8-4a9b-a844-00a067fab054]
module.database.data.cloudfoundry_service.rds: Read complete after 1s [id=dcfb1d43-f22c-42d3-962c-7ae04eda24e7]
module.sns_sms.data.cloudfoundry_space.space: Read complete after 1s [id=0b28b133-9f7b-49f1-9f34-a179cbb2549f]
module.sns_sms.cloudfoundry_service_instance.sns: Refreshing state... [id=dfa60432-eab1-456b-8a24-9a34312939ed]
module.ses_email.data.cloudfoundry_space.space: Read complete after 1s [id=0b28b133-9f7b-49f1-9f34-a179cbb2549f]
module.ses_email.cloudfoundry_service_instance.ses: Refreshing state... [id=a113b8a3-5a87-4920-85c6-00c18a8c389e]
module.redis-v70.data.cloudfoundry_space.space: Read complete after 1s [id=0b28b133-9f7b-49f1-9f34-a179cbb2549f]
module.redis-v70.cloudfoundry_service_instance.redis: Refreshing state... [id=1800d7f5-1cc7-406d-a4cb-b1d08ff1f779]
module.database.data.cloudfoundry_space.space: Read complete after 1s [id=0b28b133-9f7b-49f1-9f34-a179cbb2549f]
module.database.cloudfoundry_service_instance.rds: Refreshing state... [id=9360890e-0458-41d0-99ce-071ce7f5bef9]
module.egress-space.data.cloudfoundry_user.users["bbb35677-6906-4f8d-b230-bb4755d7886a"]: Read complete after 0s [id=74b9cc1c-d8fa-4e70-8eba-c7d66343ad9c]
module.csv_upload_bucket.data.cloudfoundry_space.space: Read complete after 0s [id=0b28b133-9f7b-49f1-9f34-a179cbb2549f]
module.csv_upload_bucket.cloudfoundry_service_instance.bucket: Refreshing state... [id=59028298-4427-4516-907d-097d99dda5c5]
module.egress-space.data.cloudfoundry_user.users["[email protected]"]: Read complete after 0s [id=ffd00a3e-aa1a-42e1-a81a-5f60420658c2]
module.egress-space.cloudfoundry_space_users.deployers: Refreshing state... [id=63a559ae-ceb6-3066-9eb0-b57347e2bece]

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration
and found no differences, so no changes are needed.

Pusher: @stvnrlly, Action: pull_request

@ccostino ccostino merged commit a054885 into production Dec 4, 2024
13 of 14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
engineering python Pull requests that update Python code
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

3 participants