Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fire zenduty alert when check fails 5 times in 5 mins #72

Merged
merged 4 commits into from
May 21, 2024

Conversation

ayazabbas
Copy link
Contributor

  • This further reduces the noise from zenduty by only alerting if a check fails 5 times within 5 minutes.
  • Subsequently we alert every 5 minutes if it continues to fail at that rate (this does not produce a new message though, it gets collated under the same incident).
  • Or if the check failed < 5 timees in the latest 5min window, the alert is resolved.
  • This should exactly match the current behaviour of datadog alerts. I ran the program for a couple hours last night and alerts aligned with the datadog alerts.

@ayazabbas ayazabbas merged commit 27a8018 into main May 21, 2024
3 checks passed
@ayazabbas ayazabbas deleted the AA_reduce-zenduty-alert-noise branch May 21, 2024 12:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants