Skip to content

Latest commit

 

History

History
247 lines (189 loc) · 7.18 KB

README.md

File metadata and controls

247 lines (189 loc) · 7.18 KB
Raw

node-kafka-streams

Build Status npm version

// suggested Node.js version: v12.16.1
npm install --save kafka-streams

const {KafkaStreams} = require("kafka-streams");

const config = require("./config.json");
const factory = new KafkaStreams(config);

const kstream = factory.getKStream("input-topic");
const ktable = factory.getKTable(/* .. */);

kstream.merge(ktable).filter(/* .. */).map(/* .. */).reduce(/* .. */).to("output-topic");

CHANGES: The latest version brings a lot of changes, please check here before updating.

API Overview

You might also like

README Overview

Prerequisites

  • Kafka broker should be version >= 0.11.x
  • Node.js should be version >= 8.x.x

A note on native mode

If you are using the native mode (config: { noptions: {} }). You will have to manually install node-rdkafka alongside kafka-streams. (This requires a Node.js version between 9 and 12 and will not work with Node.js >= 13, last tested with 12.16.1)

On Mac OS High Sierra / Mojave: CPPFLAGS=-I/usr/local/opt/openssl/include LDFLAGS=-L/usr/local/opt/openssl/lib yarn add --frozen-lockfile [email protected]

Otherwise: yarn add --frozen-lockfile [email protected]

(Please also note: Doing this with npm does not work, it will remove your deps, npm i -g yarn)

Aim of this Library

  • this is not a 1:1 port of the official JAVA kafka-streams
  • the goal of this project is to give at least the same options to a nodejs developer that kafka-streams provides for JVM developers
  • stream-state processing, table representation, joins, aggregate etc. I am aiming for the easiest api access possible checkout the word count example

Description

kafka-streams 🐙 equivalent for nodejs ✨🐢🚀✨ build on super fast 🔥 observables using most.js 🤘

ships with sinek 🙏 for backpressure

comes with js and native Kafka client, for more performance and SSL, SASL and Kerberos features

the lib also comes with a few window operations that are more similar to Apache Flink, yet they still feel natural in this api :squirrel:

overwriteable local-storage solution allows for any kind of datastore e.g. RocksDB, Redis, Postgres..

async (Promises) and sync stream operators e.g. stream$.map() or stream$.asyncMap()

super easy API :goberserk:

the lib is based on sinek, which is based on kafka-node's ConsumerGroups

Port Progress Overview

  • core structure
  • KStream base - stream as a changelog
  • KTable base - stream as a database
  • KStream & KTable cloning
  • complex stream join structure
  • advanced joins see
  • windows (for joins) see
  • flink like window operations
  • word-count example
  • more examples
  • local-storage for etl actions
  • local-storage factory (one per action)
  • KStorage example for any DB that supports atomic actions
  • backing-up local-storage via kafka
  • kafka client implementation
  • KTable replay to Kafka (produce)
  • stream for topic message production only
  • sinek implementation
  • backpressure mode for KafkaClient
  • auto-json payloads (read-map/write-map)
  • auto producer partition and keyed-message handling
  • documentation
  • API description
  • higher join & combine examples
  • embed native client librdkafka for more performance
  • SSL
  • SASL
  • Kerberos

Operator Implementations

  • map
  • asyncMap
  • constant
  • scan
  • timestamp
  • tap
  • filter
  • skipRepeats
  • skipRepeatsWith
  • slice
  • take
  • skip
  • takeWhile
  • skipWhile
  • until
  • since
  • reduce
  • chainReduce
  • forEach (observe)
  • chainForEach
  • drain
  • _zip
  • _merge
  • _join
  • _combine
  • _sample
  • throttle
  • debounce
  • delay
  • multicast
  • A description of the operators can be found here
  • Missing an operator? Feel free to open an issue 👮

Additional Operators

  • mapStringToArray
  • mapArrayToKV
  • mapStringToKV
  • mapParse
  • mapStringify
  • atThroughput
  • mapWrapKafkaPayload
  • mapToFormat
  • mapFromFormat
  • Want more? Feel free to open an issue 👮

Stream Action Implementations

  • countByKey
  • sumByKey
  • min
  • max
  • Want more? Feel free to open an issue 👮

Join Operations

Operation description

KStream Status

  • merge
  • outerJoin
  • innerJoin
  • leftJoin
  • branch

KTable Status

  • merge
  • outerJoin
  • innerJoin
  • leftJoin

KTable <-> KStream Status

  • merge
  • outerJoin
  • innerJoin
  • leftJoin

Window Operations

KStream

  • window
  • advanced window
  • rolling window

More

Can I use this library yet?

Yes.

Are we ready for production yet?

Probably, yes. 😄

Even More

Forks or Stars give motivation :bowtie: