Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat!: add new kafka backend and refactor metrics sender engine #522

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

sbengo
Copy link
Collaborator

@sbengo sbengo commented Nov 9, 2022

The SNMPCollector is currently focused and oriented to send metrics to InfluxDB, using internally specific functions of InfluxDB client to manage and send those metrics.

This PR tries to break with this dependency on metric internal management adopting the current telegraf metric definitions.
With this abstracion, a new engine is created in order to send those generic metrics to specific backends in a centralized way.

The new metric sender engine also adopts the current telegraf buffer. The metrics are now stored on the buffer instead of being written each time that a measurement produces a small set of them.

The proposed engine is changed and it is based on the current telegraf way: a ticker is defined and the metrics are flushed on several batches, reducing drastically the number of request that are being done and with the change to control the current buffer size and make operations on top the output.

Since all the metrics are stored on a top-level buffer, the send process is being done by the backend implementation. This PR redefines the current influxdb to be a backend that only performs the connection, write close.

The internal selfmon metrics -measurement, runtime, outdb stats- are refactored to follow the same generic metric definition, using, as always, the 'default' backend defined, now, as an Output.

With this behaviour, a new Kafka Backend has been created to be able to write metrics to several brokers using JSON output format. The code is copied/modified from the current telegraf.output Kafka

The SNMPCollector OutDB now is related with an Output instead of InfluxDB Server. All the management of Outputs and new Kafka Server can be done on the UI and via API. The output relation with the backend is, right now 1:1, and a Device can have only 1 output attached

BREAKING CHANGE: the metrics are now stored on internal buffer and written to the final backend based on time. The write process is being done each FlushInterval and splitted into several request based on the buffer length and the MetricsBatchSize.

BREAKING CHANGE: the InfluxDB UI component is being moved to a new section and the relation with the SNMPDevice is break and changed to an Output. The field BufferSize is being removed from API request and new DB initializtion, but mantained on old configurations to perform a migration process

fix #343

@sbengo sbengo force-pushed the feature/kafka_output branch 3 times, most recently from f022eeb to 491d76b Compare November 14, 2022 12:49
@sbengo sbengo force-pushed the feature/kafka_output branch from 491d76b to 35ad282 Compare November 24, 2022 05:21
The SNMPCollector is currently focused and oriented to send metrics to
InfluxDB, using internally specific functions of InfluxDB client to
manage and send those metrics.

This PR tries to break with this dependency on metric internal management
adopting the current telegraf metric definitions.
With this abstracion, a new engine is created in order to send those
generic metrics to specific backends in a centralized way.

The new metric sender engine also adopts the current telegraf buffer. The metrics
are now stored on the buffer instead of being written each time that
a measurement produces a small set of them.

The proposed engine is changed and it is based on the current telegraf
way: a ticker is defined and the metrics are flushed on several batches,
reducing drastically the number of request that are being done and with
the change to control the current buffer size and make operations on top
the output.

Since all the metrics are stored on a top-level buffer, the send process
is being done by the backend implementation. This PR redefines the
current influxdb to be a backend that only performs the connection, write
close.

The internal selfmon metrics -measurement, runtime, outdb stats- are
refactored to follow the same generic metric definition, using, as always,
the 'default' backend defined, now, as an Output.

With this behaviour, a new Kafka Backend has been created to be able to
write metrics to several brokers using JSON output format. The code is
copied/modified from the current telegraf.output Kafka

The SNMPCollector OutDB now is related with an Output instead of
InfluxDB Server. All the management of Outputs and new Kafka Server
can be done on the UI and via API. The output relation with the backend
is, right now 1:1, and a Device can have only 1 output attached

BREAKING CHANGE: the metrics are now stored on internal buffer and
written to the final backend based on time. The write process is being
done each FlushInterval and splitted into several request based on
the buffer length and the MetricsBatchSize.

BREAKING CHANGE: the InfluxDB UI component is being moved to a new
section and the relation with the SNMPDevice is break and changed to an
Output. The field BufferSize is being removed from API request and new
DB initializtion, but mantained on old configurations to perform a
migration process

fix #343
@sbengo sbengo force-pushed the feature/kafka_output branch from 35ad282 to c71c091 Compare November 29, 2022 15:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Feature Request] Support writing metrics to Kafka
1 participant