Improve behavior for failed data exports

[lcjohnso]

Current behavior: Exports tend to fail silently, leaving a never-changing Export is being generated. message on the Project Builder's Data Export page. Project teams are left uncertain whether they should wait (and for how long) before requesting a new export.

Proposed behavior (rough sketch; feedback and refinement welcome): Pending exports timeout after N hours (24 for workflows and subjects, longer for classifications) and/or after a liveness check on a relevant Sidekiq process comes back false. After timeout or registered failure, clear Export is being generated. message. Bonus: dead/failed jobs are re-tried.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[lcjohnso]

Reopen stale issue!

[camallen]

FWIW I figured out why our exports don't retry. It's due to our use of the https://github.com/parrish/sidekiq-congestion. When the exports are stopped and sidekiq re-queued in redis, sidekiq then tries to re-schedule them (retry) and the congestion middleware stops the job from running as it fails the congestion configs.

We need to modify the way the sidekiq congestion middleware works to have a jobs in flight identifier (that is consistent across the failures / restarts) and check for allow these in-flight jobs to restart without testing the underlying congestion data / configs.