Improve read performance

[lifeofguenter]

It seems like this plugin does not scale very well. That means, throwing CPU at logstash will not increase op/s.

I tried raising consumer_threads and pipeline.workers with no improvements.

Adding more logstash instances to the pool immediately solves the issue (though they still do not utilise CPU fully), which means its not an issue with the output itself (e.g. elasticsearch blocking) nor is there an issue with kafka itself.

Updates:

cpu underutilised:

pipeline.workers = 1
consumer_threads = 1

pipeline.workers = 1
consumer_threads = 4

pipeline.workers = 2
consumer_threads = 4

cpu utilised:

pipeline.workers = 4
consumer_threads = 4

I have not tried yet lowering the consumer_threads while keeping the pipeline.workers high. But so far it seems like pipeline.workers is the key point.