Terraform pipeline is failing for MST

[thekaveman]

Seems like we need to recycle the client secret key(s)

Failed to get existing workspaces: Error retrieving keys for Storage Account "mstcourtesycardstf": autorest/Client#Do: Preparing request failed: StatusCode=0 -- Original Error: clientCredentialsToken: received HTTP status 401 with response: 

{
  "error":"invalid_client",
  "error_description":"AADSTS7000222: The provided client secret keys for app '***' are expired. Visit the Azure portal to create new keys for your app: https://aka.ms/NewClientSecret, or consider using certificate credentials for added security: https://aka.ms/certCreds. Trace ID: xxx Correlation ID: xxx Timestamp: 2024-12-18 20:14:32Z",
  "error_codes":[7000222],
  "timestamp":"2024-12-18 20:14:32Z",
  "trace_id":"xxx",
  "correlation_id":"xxx",
  "error_uri":"[https://login.microsoftonline.com/error?code=7000222"}](https://login.microsoftonline.com/error?code=7000222%22
}

See logs: https://dev.azure.com/mstransit/courtesy-cards/_build/results?buildId=512&view=logs&jobId=ace7239b-ade7-5b52-2e3a-ab948f392fca&j=ace7239b-ade7-5b52-2e3a-ab948f392fca&t=2922d73c-5c9a-576b-ab75-a4d961dfc773

Expected behavior

terraform plan and terraform apply should succeed.

Environment

• test
• prod

Agency

• MST

Additional context

This doesn't impact the application deployment (e.g. Docker image build/push).

However, the nightly server restart (to load new data) is failing due to this, which was probably the root cause of cal-itp/benefits#2598

[angela-tran]

Refreshed the service connection credentials

• In MST's Azure DevOps, I went into Project Settings > Service connections and selected the deployer service connection
• An alert about expired credentials led me to the Certificates & secrets section of the app registration for the service connection
  • The previous client secret expired on 11/3/2024
  • I created a new client secret, but then couldn't find how to provide the service connection with the new secret
• Found these resources and followed their recommendations:
  • https://stackoverflow.com/questions/61404146/azure-devops-service-connection-expired-and-cannot-edit-renew
  • https://piotr.gg/dev-ops/how-to-update-azure-devops-service-principal-connection-once-expired.html
  • Doing a no-op save on the service connection successfully created a new client secret (and removed the expired secret and the one I created)

After doing no-op save

Also, at some point during this process, I accidentally set a Resource Group on the service connection, so this StackOverflow helped me set it back to empty so that it will work for all Resource Groups: https://stackoverflow.com/questions/72106131/azure-devops-how-to-reset-resource-group-in-edit-service-connection

[angela-tran]

The "Release" pipeline that handles restarting the app services now works:

(I manually triggered the jobs for restarting dev and test)

(I'll let the scheduled trigger handle doing production.)

The infra pipeline also now works:

[angela-tran]

#514 would be a good follow-up after we see that prod is refreshed with the existing pipeline.

[angela-tran]

Oh, also I added a calendar event to the Compiler Delivery Google Calendar for when this service connection credential expires in March 2025.