(stream/feature) deadman endpoint

[Sims24]

A community user pushed us this feature request:

In order to be alerted when a stream connector stops unexpectedly, a solution would be to offer an option to configure a deadman API endpoint. It would act as a regular heartbeat and then allow to send notifications when something goes wrong.

Some manufacturers offers such endpoint as a built-in feature but there also dedicated service like Deadman Snitch.

IMO, it would be nice to have it at the lib level so the retained solution could be use by any stream no matter what's the software vendor.

[piellick]

Hy @Sims24 ,
few examples for this feature :

• pagerduty use Deadmansnitch for this : https://www.pagerduty.com/docs/guides/dead-mans-snitch-integration-guide/
  Deadman snitch offer may way to say "hello i'am still alive" like API endpoit, email ingestion, python script ...

• OPsgenie provide an internal deadman / heartbeat function and API endpoint is available too : https://support.atlassian.com/opsgenie/docs/add-heartbeats-to-monitor-external-systems/

A simple integration could be :

• heartbeat to an API endpoint or email sending (for ingestion into the deadman)
• managing interval

pierrick

[Sims24]

Thanks @piellick ;)

[piellick]

hi,
For those looking for a way to do a deadman using engine logic and notifications, here’s a dummy script in the meantime:

#!/bin/bash

select_random() {
    printf "%s\0" "$@" | shuf -z -n1 | tr -d '\0'
}

expressions=("OK" "WARNING" "CRITICAL" )
selectedexpression=$(select_random "${expressions[@]}")
#echo "$selectedexpression"

if [ $selectedexpression == "OK" ]
then
    echo "OK"
    exit 0
elif [ $selectedexpression == "WARNING" ]
then
    echo "WARNING"
    exit 1
elif [ $selectedexpression == "CRITICAL" ]
then
    echo "CRITICAL"
    exit 2
fi

Just create a fake check through a service and send the notifs on each status change to your deadman.

Tested...do the job.