Critical Tasks & Crons should log + alert when interrupted

[mekarpeles]

Proposal

Now

• Monthly data dumps (see GitHub Action to verify successful completion of our cron jobs #6607 re: Data Dumps not auto-generated #9521)
• IA imports (e.g. Productionize Improved OpenLibrary.org ↔ Archive.org Import Process #732, related to Adding Downtime Reporting for Archive.org Imports  #5154)
• partner monthly dumps (https://github.com/internetarchive/openlibrary/blob/master/scripts/partner_batch_imports.py)
• promise item imports (https://github.com/internetarchive/openlibrary/blob/master/scripts/promise_batch_imports.py)
• resolving redirects (https://github.com/internetarchive/openlibrary/blob/master/scripts/update_stale_work_references.py)
• solr restarter (https://github.com/internetarchive/openlibrary/blob/4bc59c8306a0417218c7516672cd6b50a6277e1f/scripts/solr_restarter/index.js)

Later

• web logs

• cover archival

• solr updater

• Know that the cron / dump / service started (or was triggered)

• We want to be alerted that cron / dumps failed or succeeded

• Historical view of how often failures occur (stats.inc?)

• Look back at how often restarts attempted / failed

Justification

Problem: What problem does this proposal address & for what audience(s)? -->

Today, when crons or other critical tasks fail, we are often learning about it from patrons rather than workflows.

Impact: What's the predicted impact, how do we measure, & what does success look like?

Some of our biggest sources of value (bots that clean things up for us, like solr restarter or the bot to fix redirects) potentially don't run for weeks on end if broken.

Breakdown

Requirements Checklist

• We should be able to query over time any errors for a cron or service using statsd e.g. https://graphite.us.archive.org/render?
• We should either be pinged via slack on abc-plus when a cron job fails (or potentially create an issue for us, but let's start with slack)

[PredictiveManish]

Is this a bunch of problems together or any big project?