celery-metrics-exporter

Components

CeleryEventReceiver

this is components listening the celery events
this is where the any filter for events would live
receiver maintains a list of the exporters
on receiving a event it calls all the exporters to process the event

Exporter

this is components picks up events, processes it
the plan is to support
- Datadog stores → will start this for initial version
- Postgres stores
exporter must impl process_event which is called by the receiver
ideally exporter processing code would be started in different threads, to unblock celery event receiver.

DataDogExporter Impl

we will need to store all the events (atleast pending, received , success , failed event) to compute wait_time and run_time
the process_event function add the event to the Store object, if the event is in its final, its added to the processing_queue
- the Store has all events with task-id as key
- there is a processing_queue where task-id succeeded / failed task are tracked
once a task-id is present pushed to processing_queue , it picked by to be pushed to Datadog.

Cli

initialize the CeleryEventReceiver, Exporters
attach Exporters to CeleryEventReceiver
starts the application