Heplify server and influxDB

[jmillican61]

Hello,
I am having trouble figuring out how to use influxDB as the backing store for Heplify-server. I have seen numerous posts around the web about how to use influxdb with homer-app but none that I have found have shown how to get the data into InfluxDB, at least none that I have been able to understand.
Are there any "for dummies" type docs for getting heplify-server to insert data to InfluxDB? Or, if anyone can help me get started in the right direction I can write up a doc when I get it working if there is any interest in that.
Am I required to use prometheus between heplify-server and influxDB?
If I do get heplify-server to dump data to InfluxDB do I still need Postgres?
I am hoping to not have to install any more than necessary to be able to graph calls per minute, errors per minute, and so on. I am already using TICK for monitoring systems so I am hoping I can make use of that.
I had tried the Docker installs as a starting point but it just seems to add another level of complexity that hides a lot of what is happening and I would like to be able to dissect the pieces and understand each of them.

[lmangani]

Hello @jmillican61

Thanks for opening this topic and indeed, when we opened up Homer to be dynamic and support multiple data options we knew it might cause confusion but at the same time promote a deeper thirst for education and understanding of the connected platforms, their weaknesses and strengths, etc. Happy to start answering your questions to begin with:

1. Am I required to use prometheus between heplify-server and influxDB?
   No, but heplify-server exports its /metrics endpoint in prometheus format - which is plain text

2. how to get the data into InfluxDB
   This role in our stack is fulfilled by Telegraf, which pulls the heplify-server metrics and offers them out in many formats

3. If I do get heplify-server to dump data to InfluxDB do I still need Postgres?
   Yes if you care about packets and searching. The InfluxDB integration is purely about timeseries, statistics.

How to push HEP metrics to InfluxDB

In your specific case, I would suggest focusing on the following blocks from our default Telegraf config example

# This output will write hep statistics to influxdb
[[outputs.influxdb]] 
  urls = ["http://influxdb:8086/"] # required
  database = "homer" # required
  retention_policy = ""
  write_consistency = "any"
  timeout = "5s"
  namepass = ["hep*"]

# This input will fetch and convert statistics from heplify-server
[[inputs.prometheus]]
  ## An array of urls to scrape metrics from.
  urls = ["http://heplify-server:9096/metrics"]
  namedrop = ["go*","process*","promhttp*"]

The above configuration only requires the homer database in InfluxDB, which we can create as follows:

echo InfluxDB Retention Policy push ...;
curl -G http://influxdb:8086/query --data-urlencode 'q=CREATE DATABASE homer'
curl -G http://influxdb:8086/query --data-urlencode 'q=CREATE RETENTION POLICY "60s" ON "homer" DURATION 4w REPLICATION 1 DEFAULT'
curl -G http://influxdb:8086/query --data-urlencode 'q=drop retention policy "autogen" ON "homer"'

That's basically it. Metrics should appear as soon as traffic hits the circuit.

[jmillican61]

Thank you! I failed to mention that I have InfluxDB v2.x installed and when I run the curl's above I get errors i was able to simple create the homer bucket via the GUI though. I also had to modify the telegraf output config as follows:

[[outputs.influxdb_v2]]
urls = ["http://influxdb:8086/"] # required
organization = "APT"
token = "<read/write token for homer bucket"
bucket = "homer"
timeout = "5s"
namepass = ["hep*"]

I was able to leave the input as above.
I now see data in the homer bucket but nothing that looks like call stats. Am I still missing something?

[lmangani]

We can't guess what you are seeing :) Feel free to provide some data examples.

This said, you might just have the wrong expectations here. In a nutshell, heplify-servers is stateless and raw, it reports statistics on the protocols/methods it observes plus the statistics it receives from endpoints. Most of the useful session math is featured in the homer grafana prometheus dashboard examples, they are easy to convert to other formats as the timeseries are basically the same.

Note: If you need actual realtime session/call/registration tracking and advanced reporting, you might have to look into our sponsor HEPIC which features the same connectors but operates in a passive stateful mode (commercial)

[jmillican61]

What I am seeing, well if I go to :9090/metrics (since that is where prometheus is scraping data) i get a very long page of GO stats:

'#' HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
'#' TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 6.4071e-05
go_gc_duration_seconds{quantile="0.25"} 8.798e-05
go_gc_duration_seconds{quantile="0.5"} 0.000109647
go_gc_duration_seconds{quantile="0.75"} 0.000143826
go_gc_duration_seconds{quantile="1"} 0.000385335
go_gc_duration_seconds_sum 0.011412538
go_gc_duration_seconds_count 89

And much more
Along with conntrack stats:

'#' HELP net_conntrack_dialer_conn_attempted_total Total number of connections attempted by the given dialer a given name.
'#' TYPE net_conntrack_dialer_conn_attempted_total counter
net_conntrack_dialer_conn_attempted_total{dialer_name="alertmanager"} 0
net_conntrack_dialer_conn_attempted_total{dialer_name="default"} 0
net_conntrack_dialer_conn_attempted_total{dialer_name="heplify-server"} 1513
net_conntrack_dialer_conn_attempted_total{dialer_name="prometheus"} 1
net_conntrack_dialer_conn_attempted_total{dialer_name="remote_storage_write_client"} 57

Again and much more
Then into prometheus data:

'#' HELP prometheus_api_remote_read_queries The current number of remote read queries being executed or waiting.
'#' TYPE prometheus_api_remote_read_queries gauge
prometheus_api_remote_read_queries 0
'#' HELP prometheus_build_info A metric with a constant '1' value labeled by version, revision, branch, and goversion from which prometheus was built.
'#' TYPE prometheus_build_info gauge
prometheus_build_info{branch="HEAD",goversion="go1.15.5",revision="26d89b4b0776fe4cd5a3656dfa520f119a375273",version="2.23.0"} 1
'# 'HELP prometheus_config_last_reload_success_timestamp_seconds Timestamp of the last successful configuration reload.
'#' TYPE prometheus_config_last_reload_success_timestamp_seconds gauge
prometheus_config_last_reload_success_timestamp_seconds 1.608673266741435e+09
'#' HELP prometheus_config_last_reload_successful Whether the last configuration reload attempt was successful.
'#' TYPE prometheus_config_last_reload_successful gauge
prometheus_config_last_reload_successful 1
'#' HELP prometheus_engine_queries The current number of queries being executed or waiting.
'#' TYPE prometheus_engine_queries gauge
prometheus_engine_queries 0
'#' HELP prometheus_engine_queries_concurrent_max The max number of concurrent queries.
'#' TYPE prometheus_engine_queries_concurrent_max gauge
prometheus_engine_queries_concurrent_max 20
'#' HELP prometheus_engine_query_duration_seconds Query timings
'#' TYPE prometheus_engine_query_duration_seconds summary
prometheus_engine_query_duration_seconds{slice="inner_eval",quantile="0.5"} NaN
prometheus_engine_query_duration_seconds{slice="inner_eval",quantile="0.9"} NaN
prometheus_engine_query_duration_seconds{slice="inner_eval",quantile="0.99"} NaN
prometheus_engine_query_duration_seconds_sum{slice="inner_eval"} 0.005177317000000001
prometheus_engine_query_duration_seconds_count{slice="inner_eval"} 24
prometheus_engine_query_duration_seconds{slice="prepare_time",quantile="0.5"} NaN
prometheus_engine_query_duration_seconds{slice="prepare_time",quantile="0.9"} NaN
prometheus_engine_query_duration_seconds{slice="prepare_time",quantile="0.99"} NaN
prometheus_engine_query_duration_seconds_sum{slice="prepare_time"} 0.001836631
prometheus_engine_query_duration_seconds_count{slice="prepare_time"} 24

Which goes on for quite a lot of lines.

what I was hoping to be able to do is insert the hep packet data into InfluxDB. I can query the data from Postgresql with a query such as:

select count(*)/60 from hep_proto_1_registration where raw like '%REGISTER%' and create_date > NOW() - INTERVAL '1' HOUR and protocol_header->>'captureId' = '1983';

With this I get the number of registers in 1 minute over the course of the last hour. This is the type of data I am hoping to be able to throw up on a dashboard in either chronograf or grafana. Along with Invites, byes, and so on.
Am I barking up the wrong tree?

Side Note: I definitively appreciator the fast responses since I see by your profile you are possibly in Amsterdam. if so What the heck are you doing a wake? :-) I am in Alaska (GMT -9).

[lmangani]

I would say the namepass rule is not working as intended, as it should only pass heplify metrics and drop all of the above....

[jmillican61]

Ok. I will look into that to see if I can find why. Thanks again for all your help.