0

I'm using Apache Kafka 0.8.2.1, planning to upgrade my application to use Apache kafka 1.0.0. While I inspect about Kafka Streams, I got some question about difference between KafkaConsumer and KafkaStreams.

Basically, KafkaConsumer have to consume from broker using polling method. I can specify some duration while polling, and whenever I got ConsumerRecored I can handle it to product some useful information. KafkaStream, on the other hand, I don't have to specify any polling duration but just call start() method.

I know that KafkaConsumer basically used to consume literally, from broker and KafkaStreams can do various thing like Map-Reduce or interact with database, even re-produce to other kafka or any other systems.

So, there is my question. Is there any difference between KafkaConsumer and KafkaStream basically(in other words, When it comes to level of apache kafka library.)?

WillSmith
  • 62
  • 1
  • 11

2 Answers2

3

Yes, there is the difference between Kafka Consumer and Kafka Streams.

Kafka Consumer can be used at receiving end to receive data and process for future computation(based on topic and partition) Kafka Streams API to store and distribute, in real-time, published content to the various applications and systems that make it available to the readers.

Sandish Kumar H N
  • 312
  • 2
  • 11
2

As you've said, they offer different functionalities:

  • KafkaStreams allows to perform complex processing on records
  • KafkaConsumer allows to receive records from a Kafka Cluster

KafkaStreams uses regular KafkaConsumers and KafkaProducers clients under the cover in order to retrieve records and send the results of processing to the brokers. It uses predefined values for many configurations but still exposes a lot of client configurations.

KafkaStreams is a regular (although pretty advanced) Kafka application using the Kafka clients (Consumer and Producer). Its APIs allow higher level applications to focus on the business logic and not on the Kafka details.

Also being part of the Apache Kafka distribution, it's using best practices and tricks to make the most of Kafka.

Mickael Maison
  • 25,067
  • 7
  • 71
  • 68
  • 2
    The biggest difference might be, that Kafka Streams allows you to build fault-tolerant stateful applications while a plain KafkaConsumer does not provide any support for this. – Matthias J. Sax Feb 05 '18 at 18:07