0

I am new to Kafa but have experience with ActiveMQ. When reading about Kafka I understand that a topic can have multiple partitions that can be produced to in a "round-robin" technique. I also understand that you can only have one consumer per topic partition.

What I cannot understand so far is the benefit in using multiple partitions with a single consumer on each over using a single ActiveMQ queue and having multiple consumers (threads).

Let me attempt to demonstrate.

Kafka Produce messages 1,2,3 to a topic with 3 partitions. Partition 1 would get message 1, partition 2 would get message 2 and partition 3 would get message 3. I would then potentially need 3 consumers (with configuration) to get the 3 messages concurrently.

ActiveMQ Produce messages 1,2,3 to a queue. I could have a single message listener (one MDB for example with a single configuration) that would start in a thread for each message on the queue.

It appears to me that I need to do additional work and configuration to consume the 3 messages from Kafka (3 partitions) over the work required for the single ActiveMQ queue.

Am I missing something?

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Aidan
  • 3
  • 3
  • If you want a fair comparison, why would you have 3 Kafka partitions? Also, if you set the same message key in Kafka, then all records go to one partition, not round-robin – OneCricketeer Sep 10 '18 at 22:23
  • "only one consumer per partition" is also slightly false. You can have many consumers, but only one *consumer group* assigns a consumer *in that group* to any one partition. – OneCricketeer Sep 10 '18 at 22:24
  • BTW, https://stackoverflow.com/questions/44792604/what-is-the-difference-between-apache-kafka-vs-activemq and other topics https://stackoverflow.com/questions/tagged/apache-kafka+activemq?sort=votes – OneCricketeer Sep 11 '18 at 06:29
  • Understood. So if I had one type of message and 3 partitions, do I need to create 3 consumers and configure each of them to consume a specific partition? I would have thought this would duplicate code. Again, early learning on Kafka so apologies for the simple questions :-) – Aidan Sep 11 '18 at 19:04
  • Also, doesn't that mean that a Kafka partition is really a queue inside a topic? – Aidan Sep 11 '18 at 19:06
  • A partition is indeed a queue within a topic. And yes, consumers can only be assigned a single partition, but that's not something you need to assign manually, as the consumer group protocol within Kafka will do that for you. Though, that's just the consumer API, and in a more general case, you can use the higher level Kafka Streams API to do quick transformations and filters – OneCricketeer Sep 12 '18 at 00:37

0 Answers0