Gollum is a n:m multiplexer that gathers messages from different sources and broadcasts them to a set of destinations.
There are a few basic terms used throughout Gollum:
- "Consumers" read data from other services
- "Producers" write data to other services
- "Streams" route data between consumers and producers
- A "message" is a set of data passed between consumers and producers
- "Formatters" can transform the content of messages
- "Filters" can block/pass messages based on their content
Writing a custom plugin does not require you to change any additional code besides your new plugin file.
A how-to-use documentation can be found on read the docs. Developers should use the godoc pages to get started. Plugin documentation is generated from the plugin source code. So if you feel that something is missing a look into the code may help.
Console
read from stdin.File
read from a file (like tail).Http
read http requests.Kafka
read from a Kafka topic.Kinesis
read from a Kinesis stream.Profiler
Generate profiling messages.Proxy
use in combination with a proxy producer to enable two-way communication.Socket
read from a socket (gollum specific protocol).Syslogd
read from a socket (syslogd protocol).SystemD
read from the SystemD journal.
Console
write to stdin or stdout.ElasticSearch
write to elasticsearch via http/bulk.File
write to a file. Supports log rotation and compression.Firehose
write data to a Firehose stream.HTTPRequest
HTTP request forwarder.InfluxDB
send data to an InfluxDB server.Kafka
write to a Kafka topic.Kinesis
write data to a Kinesis stream.Null
like /dev/null.Proxy
two-way communication proxy for simple protocols.S3
write data to Amazon S3 stream.Scribe
send messages to a Facebook scribe server.Socket
send messages to a socket (gollum specific protocol).Spooling
write messages to disk and retry them later.Websocket
send messages to a websocket.
Broadcast
send to all producers in a stream.Random
send to a random producer in a stream.RoundRobin
switch the producer after each send in a round robin fashion.Route
convert streams to one or multiple others
Base64Encode
encode messages to base64.Base64Decode
decode messages from base64.Clear
clears a messageCollectdToInflux08
convert CollectD 0.8 data to InfluxDB compatible values.CollectdToInflux09
convert CollectD 0.9 data to InfluxDB compatible values.CollectdToInflux10
convert CollectD 0.10 data to InfluxDB compatible values.ExtractJSON
extracts a single field from a JSON object.Envelope
add a prefix and/or postfix string to a message.Forward
write the message without modifying it.Hostname
prepend the current machine's hostname to a message.Identifier
hash the message to generate a (mostly) unique id.JSON
write the message as a JSON object. Messages can be parsed to generate fields.ProcessJSON
Modify fields of a JSON encoded message.ProcessTSV
Modify fields of a TSV encoded message.Runlength
prepend the length of the message.Sequence
prepend the sequence number of the message.SplitToJSON
tokenize a message and put the values into JSON fields.StreamName
prepend the name of a stream to the payload.StreamRevert
route a message to the previous stream (e.g. after it has been routed).StreamRoute
route a message to another stream by reading a prefix.Timestamp
prepend a timestamp to the message.
All
lets all message pass.JSON
blocks or lets json messages pass based on their content.None
blocks all messages.Rate
blocks messages that go over a given messages per second rate.RegExp
blocks or lets messages pass based on a regular expression.Stream
blocks or lets messages pass based on their stream name.
Installation from source requires the installation of the Go toolchain.
Gollum supports the Go 1.5 vendor experiment that is automatically enabled when using the provided makefile.
With Go 1.6 and later you can also use go build
directly without additional modifications.
Builds with Go 1.4 or earlier versions are not officially supported and might require additional steps and modifications.
$ make
$ ./gollum --help
You can use the make file coming with gollum to trigger cross platform builds.
Make will produce ready to deploy .zip files with the corresponding platform builds inside the dist folder.
To test gollum you can make a local profiler run with a predefined configuration:
$ gollum -c config/profile.conf -ps -ll 3
By default this test profiles the theoretic maximum throughput of 256 Byte messages.
You can enable different producers in that config to test the write performance of these producers, too.
Configuration files are written in the YAML format and have to be loaded via command line switch. Each plugin has a different set of configuration options which are currently described in the plugin itself, i.e. you can find examples in the GoDocs.
Use a given configuration file.
Print this help message.
Set the loglevel [0-3]. Higher levels produce more messages as in 0=Errors, 1=Warnings, 2=Notes, 3=Debug.
Port to use for metric queries. Set 0 to disable.
Number of CPUs to use. Set 0 for all CPUs.
Write the process id into a given file.
Write CPU profiler results to a given file.
Write heap profile results to a given file.
Write msg/sec measurements to log.
Print detailed version report and quit.
Test a given configuration file and exit.
Write trace results to a given file.
Print version information and quit.
The easiest way to install go is by using homebrew:
brew install go
Download Go from the golang website and unzip it to e.g. /usr/local/go.
You have to set the GOROOT environment variable to the folder you chose:
export GOROOT=/usr/local/go
If you do not already have a GOPATH set up you need to create one.
The location is free of choice, we prefer to put it into each users home folder:
mkdir -p ~/go
export GOPATH=$(HOME)/go
You can download gollum via go get github.com/trivago/gollum
or clone it directly into your GOPATH.
If you choose this way you need to download your dependencies directly from that folder
mkdir -p $(GOPATH)/src/github.com/trivago
cd $(GOPATH)/src/github.com/trivago
git clone https://github.com/trivago/gollum.git
cd gollum
Building gollum is as easy as make
or go build
.
When using Go 1.5 make sure to enable the go vendor experiment by setting export GO15VENDOREXPERIMENT=1
or use make
.
If you want to do cross platform builds use make all
or specify one of the following platforms instead of "all":
current
build for current OS (default)freebsd
build for FreeBSDlinux
build for Linux x64mac
build for MacOS Xpi
build for Linux ARMwin
build for Windows
There are also supplementary targets for make:
clean
clean all artifacts created by the build processtest
run unittestsvendor
install gvt and update all dependenciesaws
build for Linux x64 and generate an Elastic Beanstalk package
If you want to use native plugins (contrib/native) you will have to enable the corresponding imports in the file contrib/loader.go. Doing so will disable the possibility to do cross-platform builds for most users.
The repository contains a Dockerfile
which enables you to build and run gollum inside a Docker container.
$ docker build -t trivago/gollum .
$ docker run -it --rm trivago/gollum -c config/profile.conf -ps -ll 3
To use your own configuration you could run:
$ docker run -it --rm -v /path/to/config.conf:/etc/gollum/gollum.conf:ro trivago/gollum -c /etc/gollum/gollum.conf
This project is released under the terms of the Apache 2.0 license.