6

I have worked on some Kafka stream application and Kafka consumer application. In the end, Kafka stream is nothing but consumer which consumes real-time events from Kafka. So I am not able to figure out when to use Kafka streams or why we should use Kafka streams as we can perform all transformation on the consumer end.

I want to understand the main difference between Kafka stream and Kafka consumer as implementation wise and how to make a decision about what we should use in different use cases.

Thanks in advance for answers.

Robin Moffatt
  • 30,382
  • 3
  • 65
  • 92
suraj shinde
  • 198
  • 4
  • 10
  • Check this out https://stackoverflow.com/questions/44014975/kafka-consumer-api-vs-streams-api/58758670?noredirect=1#comment104553475_58758670 – Nitin Dec 04 '19 at 17:05
  • Duplicate of https://stackoverflow.com/questions/44014975/kafka-consumer-api-vs-streams-api. – miguno Dec 05 '19 at 11:08
  • 1
    Does this answer your question? [Kafka: Consumer API vs Streams API](https://stackoverflow.com/questions/44014975/kafka-consumer-api-vs-streams-api) – miguno Dec 05 '19 at 11:08

2 Answers2

6

It's a question about "easy of use" (or simplicity) and "flexibility". The two "killer features" of Kafka Streams, compared to plain consumer/producer are:

  • built-in state handling, and
  • exactly-once processing semantics.

Building a stateful, fault-tolerant application or using Kafka transactions with plain consumers/producers is quite difficult to get right. Furthermore, the higher level DSL provides a lot of built-in operators that are hard to build from scratch, especially:

  • windowing and
  • joins (stream-stream, stream-table, table-table)

Another nice feature is punctuations.

However, even if you build a simple stateless application, using Kafka Streams can help you significantly to reduce you code base (ie, avoid boilerplate code). Hence, the recommendation is, to use Kafka Streams when possible and only fall back to consumer/producer if Kafka Streams is not flexible enough for your use case.

Matthias J. Sax
  • 59,682
  • 7
  • 117
  • 137
4

It's different ways to do the same thing, with different levels of abstraction and functionality.

Here's a side-by-side comparison of doing the same thing (splitting a string into two separate fields) in Kafka vs in Kafka Streams (for good measure it shows doing it in ksqlDB too)

Robin Moffatt
  • 30,382
  • 3
  • 65
  • 92