vmq_kafka: a VerneMQ plugin that sends all published messages to Apache Kafka
- You need to recompile vernemq with support to the brod OTP application: Brod App
- Modify the rebar.config file in the top directory of VerneMQ and add the brod application as dependancy
- Example of a modified rebar.config is available here: rebar.config.vernemq
- In order to have the brod app loadad and configured along with VerneMQ server you need to:
- Create your own version of advanced.config where you specify the Kafka host and port (default is localhost:9092)
- Copy the file advanced.config in the VerneMQ config directory (same directoru as vernemq.conf)
- Recompile VerneMQ as described in the broker documentation, in the section build from sources
We assume that the plugin is installed.
The plugin can be started at broker start by adding this line:
plugins.vmq_kafka = on
The Brod App client (OTP service name) can be changed (needs to match the one in advanced.config):
plugins.vmq_kafka.brod_client = my_brod_client_name
The number of partitions can be changed:
plugins.vmq_kafka.num_partitions = 5
The Kafka topic where all the mqtt pubs are ingested can be changed:
plugins.vmq_kafka.kafka_topic = my_kafka_topic
- There is a running Kafka development cluster with anonymous access (example here)
- We assume that the configured Kafka topic exist with a number of partitions that matches num_partitions
- Important: you need to ensure that the kafka topic (and partitions) are prepared in kafka before you start the vernemq cluster
- In case you need to alter partitions and topic in Kafka you need to restart VerneMQ
- Minumum for num_partitions is 1
- If num_partitions is more than 1 the producers will send the message to a randomized partition in the interval [0, num_partitions-1]
- The Kafka key value is now empty
- The mqtt message is encoded in protobuf using this schema
You must have a recent version of Erlang/OTP installed (it's recommended to use the same one VerneMQ is compiled with). To compile run:
rebar3 compile
Then enable the plugin using:
vmq-admin plugin enable --name vmq_kafka --path <PathToYourPlugin>/vmq_kafka/_build/default
The <PathToYourPlugin>
should be accessible by VerneMQ (file permissions).
At the moment this plugin is in development stage and not ready for a production environment. Please refer to VerneMQ manual on how to package the plugin for a production environment.
To redclawtech for creating this project that helps creating a proper generic vernemq plugin code.