Skip to content

A VerneMQ plugin that sends all published messages to Apache Kafka

License

Notifications You must be signed in to change notification settings

crisrise/vmq_kafka

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

6 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

VerneMQ Kafka Plugin

vmq_kafka: a VerneMQ plugin that sends all published messages to Apache Kafka

Prerequisites

  1. You need to recompile vernemq with support to the brod OTP application: Brod App
    • Modify the rebar.config file in the top directory of VerneMQ and add the brod application as dependancy
    • Example of a modified rebar.config is available here: rebar.config.vernemq
  2. In order to have the brod app loadad and configured along with VerneMQ server you need to:
    • Create your own version of advanced.config where you specify the Kafka host and port (default is localhost:9092)
    • Copy the file advanced.config in the VerneMQ config directory (same directoru as vernemq.conf)
  3. Recompile VerneMQ as described in the broker documentation, in the section build from sources

Configuration variables in vernemq.conf

We assume that the plugin is installed.

The plugin can be started at broker start by adding this line:

plugins.vmq_kafka = on

The Brod App client (OTP service name) can be changed (needs to match the one in advanced.config):

plugins.vmq_kafka.brod_client = my_brod_client_name

The number of partitions can be changed:

plugins.vmq_kafka.num_partitions = 5

The Kafka topic where all the mqtt pubs are ingested can be changed:

plugins.vmq_kafka.kafka_topic = my_kafka_topic

Important notes:

  1. There is a running Kafka development cluster with anonymous access (example here)
  2. We assume that the configured Kafka topic exist with a number of partitions that matches num_partitions
    • Important: you need to ensure that the kafka topic (and partitions) are prepared in kafka before you start the vernemq cluster
    • In case you need to alter partitions and topic in Kafka you need to restart VerneMQ
  3. Minumum for num_partitions is 1
  4. If num_partitions is more than 1 the producers will send the message to a randomized partition in the interval [0, num_partitions-1]
  5. The Kafka key value is now empty
  6. The mqtt message is encoded in protobuf using this schema

Usage

You must have a recent version of Erlang/OTP installed (it's recommended to use the same one VerneMQ is compiled with). To compile run:

rebar3 compile

Then enable the plugin using:

vmq-admin plugin enable --name vmq_kafka --path <PathToYourPlugin>/vmq_kafka/_build/default

The <PathToYourPlugin> should be accessible by VerneMQ (file permissions).


Deploy in production

At the moment this plugin is in development stage and not ready for a production environment. Please refer to VerneMQ manual on how to package the plugin for a production environment.


Thanks

To redclawtech for creating this project that helps creating a proper generic vernemq plugin code.

About

A VerneMQ plugin that sends all published messages to Apache Kafka

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages