Create concurrent Kafka consumers in Camel-Kafka

Question

I'm using camel-kafka version 2.14.3 . Below is the kafka URI :

<from uri="kafka:{brokerlist}?topic={topic-name}&amp;zookeeperHost={zookeeperHost}&amp;zookeeperPort={zookeeperPort}&amp;groupId={groupId-name}&amp;consumerStreams=2" />

note that I have used consumerStream=2 in URI options. But when I publish multiple messages to the topic topic-name at once (all to the same partition), the kafka consumer receives those messages sequentially. How can receives those messages parallely?

I'm looking for a solution like below :

<from uri="ibm_bean_name:queue_name?concurrentConsumers=2" />

is what I use to read concurrently from an ibm MQ

There is a property `consumersCount` for camel kafka – Debjit Sep 24 '22 at 23:52 — Debjit, Sep 24 '22 at 23:52

score 1 · Accepted Answer · answered Apr 21 '20 at 09:16

1

The partition is Kafka's unit of parallelism. Each partition will be assigned to a single consumer in a given consumer-group. You said you are producing to a single partition. That means all the messages will go to a single consumer, regardless of how many consumers you have in the group. If you want the messages to go to different consumers, you need to produce to different partitions.

answered Apr 21 '20 at 09:16

Rigel Bezerra de Melo

101
3

Thanks for the answer. Does that mean, I cannot have multiple consumer threads of same consumer group reading concurrently on the same partition? – vineeth kumar Apr 21 '20 at 09:28
1

You cannot. That could lead to out-of-order consumption, which is an important guarantee from Kafka. See https://stackoverflow.com/questions/25896109/in-apache-kafka-why-cant-there-be-more-consumer-instances-than-partitions – Rigel Bezerra de Melo Apr 21 '20 at 09:34
Thanks for the reply, that answers my question. Just to confirm, to have parallelism using consumerStreams option, I should be publishing messages to different partitions right? – vineeth kumar Apr 21 '20 at 09:58

score 1 · Answer 2 · answered Apr 21 '20 at 10:09

If you have N partitions, then you can have up to N consumers within the same consumer group each of which reading from a single partition. When you have less consumers than partitions, then some of the consumers will read from more than one partition. Also, if you have more consumers than partitions then some of the consumers will be inactive and will receive no messages at all.

If you have one consumer per partition, then some of the partitions might receive more messages and this is why some of your consumers might be idle while some others might still processing some messages. Note that messages are not always inserted into topic partitions in a round-robin fashion as messages with the same key are placed into the same partition.

Create concurrent Kafka consumers in Camel-Kafka

2 Answers2