When will there be support for Kafka Streams?

[jcmuller]

Description

Currently evaluating what language and library to use for a brand new application, and we would like to not be constrained by picking a language that runs on the JVM. The best option so far appears to be a library built atop librdkafka, but for our application, leveraging Kafka Streams would make a lot of sense. Are there going to be supported any time soon?

[edenhill]

Ping @miguno

[jadbox]

Getting Kafka Streaming KTable support in librdkafka would be huge for expanding these exciting features beyond only JVM-based languages.

[solsson]

Even basic support for in-memory state based on log compaction style topic would be great, without the filtering etc. Could expose a leveldb/rocksdb/etc query API. Anyhow it sounds to me like separate projects.

I found https://github.com/bitbouncer/kspp. @skarlsson can you comment on the status of the project?

[skarlsson]

@solsson kspp is a work-in-progress but it's deployed in production. It started as a hobby project of mine (of course inspired by java version) and then got used by some clients (as internal forks). It supports avro, binary, text or json. Avro support using confluent schema registry + a hackish addon to the official avro "avrogencpp". The in-memory left-joins and flatmap, re-partition is really useful and very fast. A big difference is that it does not do the "kafka streams auto rebalancing" but relies on serving static groups of partitions (one or many) - you need something else for orchestration (mesos or kubernetes...) - the idea was to get rid of rebalancing problem. Finally there are a simple built in metrics generation that can output stats to a kafka topic in influxdb format. (use telegraf to get that topic to influxdb). Summary: Lots of missing features but usable. Quite easy to extend and of course - help appreciated

[JRGranell]

Hi, I wanted to ask if streams were now supported as it has been some time since this question was asked?

We're using node-kafka-streams which wraps librdkafka and we have an issue where our KTables are not being persisted across re-starts. I.e. its in-memory only and the state is not stored by Kafka in the broker. This issue raised with in that repo nodefluent/kafka-streams#86.

It's possible it's either a configuration or user error our on our end, or that lib doesn't support it, but I wanted to try and confirm if librdkafka supports streaming behaviour in the same way the Java implementation does?

Thanks in advance, and appreciate any help.

[edenhill]

@JRGranell librdkafka does not implement Kafka Streams and there are no plans for this either.
What librdkafka provides to node-kafka-streams are simply the underlying producer and consumer clients, any issues with streams will need to be handled by the nodefluent project.

[JRGranell]

@edenhill thanks for the quick reply,