Prepare for Census 100,000 message send

[ecayer]

The team is preparing for a first “large” send of (100,000) messages. Accepting this partner is in line with the product strategy of supporting high-volume partners in order to ultimately secure paid partnerships.

The team expressed a high level of confidence that with thoughtful batching, the new partner would be successful.

High level tasks

• (Cathy & Stvn) talk to Census about batching. Find out what impact of carrier throttling would be. Also potentially review the actual message to ensure it’s one message part
• (Carlo?) Watch for what memory does during this week (possibly look for prior spikes and see how high they went / how many messages were going out)
  • Update 11/25/24 - Carlo sent out a cloud.gov support request asking for metrics of our database the application
• (Ken) Load testing on prod - working through details on Slack
  • This was conducted on 11/22/24
• (?) The day before send-day, make final deployments and up the memory on prod.
• (?) Clarify any extraordinary “on-call” expectations (which I think are “none” :-) )
• (?) Think through the possibilities for avoiding/detecting/mitigating carrier throttling (did not get to it in the session, but clearly high on people’s minds!)
• (?) Give AWS and cloud.gov a heads-up

Specific engineering tasks

• Review and improve/correct our Celery task retry logic notifications-api#1391
• Investigate the sqlalchemy.exc.IntegrityError exceptions from large batch send test and report findings notifications-api#1430
• Research adjusting the prefork setting for Celery and write an ADR proposing adjustments notifications-api#1432
• Add carrier name and check on carrier error response in reports notifications-api#1426
• Adjust the worker-max-tasks-per-child setting from 200 to 500 for Celery notifications-api#1431
• Research the proper amount of workers and RAM (memory) per worker for Celery notifications-api#1390
• Upgrade our production database instance to a higher tier plan notifications-api#1436

Addressing errors/false positives in New Relic

• Investigate sqlalchemy.exc:NoResultFound, which could be related to the botocore.errorfactory:NoSuchKey errors notifications-api#1396
• Research why botocore.errorfactory:NoSuchKey errors continue to be thrown and what can be done to prevent them notifications-api#1392