1.03.5 Kafka Performance Issue Analysis & Investigation - Part2.5 - Research Streaming API capabilities and support for Node

[mdebarros]

Goal:

1. As a contributor
   I would like to perform Kafka Performance Issue Analysis & Investigation
   so that a viable solution can be identified and recommended for implementation
2. As an FSP
   I would want the development team to do analysis regarding Kafka performance issue
   so that transfers can be performant

Tasks:

• Kafka Performance Environment - Research Exactly-once Semantics via the Streaming API and identify if NodeJs is able to leverage on this. [ @mdebarros ]

---

Follow-up story - #362

---

Acceptance Criteria:

• Document Research Findings

Dependencies:

• n/a

Accountability:

• Owner @mdebarros
• QA/Review: Design Authority

[mdebarros]

Findings

Native Kafka Streaming API Support

There is very little support for native Kafka Streaming API capabilities within Node-rdkafka due to the dependency on the Librdkafka which has not implemented this functionality.
(Ref: confluentinc/librdkafka#1248)

Note that Node-rdkafka does expose a Streaming API, but it does not support the same capabilites as a Native Kafka Streaming API (which requires EOS support for the processing.guarantee=exactly_once config). It is merely a Stream wrapped around the existing Librdkafka implementation at a Node level. It is also important to note that this Streaming API is not as fast as the Standard API which has been implemented in the Common Streaming Library. (Ref: https://github.com/Blizzard/node-rdkafka/blob/master/lib/kafka-consumer-stream.js).

Exactly-Once Semantic Support (EOS)

EOS capabilities are not supported by Librdkafka (and thus Node-rdKafka). However Librdkafka hope to have it implemented by Q4 2018. (Ref: confluentinc/librdkafka#1308).

Common Streaming Library support for EOS

In the Common Streaming Library we have followed a similar approach to the following post: SOHU-Co/kafka-node#548 by implementing the Async Queue/DeQueue + Consumer Pause when processing a message. This however doesn't provide the Exactly-Once semantic without some additional configurations & implementation done by the Client Implementation which is covered in the next section.

Mojaloop handles EOS by the following approach:

1. Async Queue/Dequeue and pausing the Kafka Consumer while processing consumed messages ~ (capability provided by the Common Streaming Library)
2. Enforcing a single Partition to ensure the ordering of messages by a single Consumer ~ ( done by the configuration of the Topic)
3. Commit the Kafka Index after each message has been processed (regardless of batches or single messages).

Take note that the point 3. will not be applicable when using auto-commits or only committing the index at the end of processing a batch of messages. It is up to the Client implementation to ensure that point 3. is implemented to ensure the Exactly-Once semantic.