Single rider is unable to verify with their MST Courtesy Card

[thekaveman]

Do not put sensitive information in this issue or Slack. Use a file on Google Drive with access restricted.

Slack thread: https://cal-itp.slack.com/archives/C037Y3UE71P/p1734391497975419

Severity: Low

Initiate

1. Create an issue from this template.
2. Declare an incident in the relevant Slack channel, such as #benefits-general.
3. Delegate subsequent tasks.

Assess

• Determine the impact.
• Assign the severity above:
  • High: Possible/confirmed breach of sensitive information, such as production system credentials or personally-identifiable information (PII)
  • Medium: Full Benefits downtime for more than 30 minutes
  • Low: Partial service degredation, zero-day vulnerabilities, data loss

Remediate

• Post in the Slack thread when the incident has been resolved.
• Retain any relevant materials.

Follow-up

• For Medium/High incidents, write an incident report. Past examples.
  1. Write a draft.
     • Link to relevant Slack messages, etc.
  2. Get thumbs-up from those involved in the incident.
  3. Share with relevant stakeholders.
• Create issues for follow-up tasks

[thekaveman]

We've resolved the immediate issue for the rider. The Azure client secret that allows Azure DevOps pipelines to connect to the Azure Subscription where the App Service etc. is defined, had expired. This meant that the job that restarts the server (causing a fresh data load) was failing, and the rider's information never made it into the database.

The client secret has been refreshed and we have confirmed the data load works as expected. The rider was able to enroll after MST notified them to try again.

We're putting some additional monitoring in place so that we can get ahead of a failed data load in the future.