feature: add prometheus collector, to collect metrics.

[laixintao]

Those metrics are collected:

• workers
• jobs
• schedulers

close #503

[laixintao]

the metrics will be something like this:

# HELP rq_jobs RQ messages Count
# TYPE rq_jobs gauge
rq_jobs{queue="default"} 0.0
# HELP rq_registry_count_total RQ Registry Count
# TYPE rq_registry_count_total counter
rq_registry_count_total{queue="default",state="deferred"} 0.0
rq_registry_count_total{queue="default",state="failed"} 2.0
rq_registry_count_total{queue="default",state="finished"} 5.0
rq_registry_count_total{queue="default",state="scheduled"} 0.0
rq_registry_count_total{queue="default",state="started"} 0.0
# HELP rq_workers RQ workers
# TYPE rq_workers gauge
rq_workers{name="b5834df2de0f466b8f488c187bf95d16",queues="default",state="idle"} 1.0
rq_workers{name="dc091f1a9c414442b19a394b4382712b",queues="default",state="idle"} 1.0
rq_workers{name="ebfe67af68df46c78bde34b37737fbf7",queues="default",state="idle"} 1.0
rq_workers{name="620bb27fe0434ef580441e9314c5eaf1",queues="default",state="idle"} 1.0
# HELP rq_worker_job_count_total RQ job count per worker
# TYPE rq_worker_job_count_total counter
rq_worker_job_count_total{state="successful",worker_name="b5834df2de0f466b8f488c187bf95d16"} 48.0
rq_worker_job_count_total{state="failed",worker_name="b5834df2de0f466b8f488c187bf95d16"} 0.0
rq_worker_job_count_total{state="successful",worker_name="dc091f1a9c414442b19a394b4382712b"} 38.0
rq_worker_job_count_total{state="failed",worker_name="dc091f1a9c414442b19a394b4382712b"} 1.0
rq_worker_job_count_total{state="successful",worker_name="ebfe67af68df46c78bde34b37737fbf7"} 45.0
rq_worker_job_count_total{state="failed",worker_name="ebfe67af68df46c78bde34b37737fbf7"} 0.0
rq_worker_job_count_total{state="successful",worker_name="620bb27fe0434ef580441e9314c5eaf1"} 30.0
rq_worker_job_count_total{state="failed",worker_name="620bb27fe0434ef580441e9314c5eaf1"} 0.0
# HELP rq_worker_workings_seconds_total RQ worker work seconds total
# TYPE rq_worker_workings_seconds_total counter
rq_worker_workings_seconds_total{worker_name="b5834df2de0f466b8f488c187bf95d16"} 64.613818
rq_worker_workings_seconds_total{worker_name="dc091f1a9c414442b19a394b4382712b"} 123.125804
rq_worker_workings_seconds_total{worker_name="ebfe67af68df46c78bde34b37737fbf7"} 61.776564
rq_worker_workings_seconds_total{worker_name="620bb27fe0434ef580441e9314c5eaf1"} 145.002066
# HELP rq_worker_current_working_seconds_gauge_total RQ worker work seconds for current job in seconds
# TYPE rq_worker_current_working_seconds_gauge_total counter
rq_worker_current_working_seconds_gauge_total{worker_name="b5834df2de0f466b8f488c187bf95d16"} 0.0
rq_worker_current_working_seconds_gauge_total{worker_name="dc091f1a9c414442b19a394b4382712b"} 0.0
rq_worker_current_working_seconds_gauge_total{worker_name="ebfe67af68df46c78bde34b37737fbf7"} 0.0
rq_worker_current_working_seconds_gauge_total{worker_name="620bb27fe0434ef580441e9314c5eaf1"} 0.0
# HELP rq_sheduler_jobs_in_next_hour_gauge RQ scheduled jobs in 1 hour
# TYPE rq_sheduler_jobs_in_next_hour_gauge gauge
rq_sheduler_jobs_in_next_hour_gauge 1.0

[selwin]

Sorry for the late reply. Enabling more ways to collect stats is a great idea. Are you open to the idea of implementing stat collection mechanism in RQ itself (instead of Django-RQ)? If we create this in RQ, other projects can benefit from this.

The stat collection mechanism should look something like this:

stats = WorkerStats('name', redis)
print(stats.job_count)
print(stats.total_working_time)

stats = QueueStats('name', redis)
print(stats.job_count)
...

Once we have these stats primitives, it will be easy for other people to use/integrate with prometheus or other monitoring tools.

[zifter]

Actually there is ready to use exporter. It's better to use it.
https://github.com/mdawar/rq-exporter

[laixintao]

@zifter rq-exporter's metrics are too few, not enough for me. besides, I don't want to run another process just for collecting metrics.