fix notification insert retry logic

[terrazoon]

Description

There was flaw in the logic that inserts notifications into the db. In the event of a duplicate insert, we were supposed to quit trying immediately. However, because IntegrityError is a subclass of SQLAlchemyError we ended falling through the catch phase and activating the retry logic. So we ended up getting 5 copies of the duplicate error in the logs, and because we hit the SQLAlchemyError the whole exception stack trace got printed, which resulted in millions of log lines when we ran a 25k send.

Fix the logic so we exit abruptly on a duplicate insert. This short circuits the retry logic and results in only one warning level log line. This, in combination with having each worker support 500 tasks instead of 200 tasks, should reduce the volume of logging from retries by over 90%. Note that there is a separate category of log pollution for 'NoSuchKey' that is written up in a separate ticket and not directly related to this issue (although eliminating the retries may end up helping).

Security Considerations

N/A

[A-Shumway42]

Great catch! Looks good.

[ccostino]

Thanks, @terrazoon!