Supposing we have 4 consumers with the same group-id and one topic including 3 partitions. If a producer post a message to partition-1, then which consumer will get this message?
Asked
Active
Viewed 334 times
1
Neo
- 2,196
- 5
- 32
- 58
-
1First, the fourth consumer in your example will do nothing because if the number of consumers is greater than the number of partitions some consumers will be idle and will get no partitions to consume. Second, please see this answer on how rebalancing works -http://stackoverflow.com/questions/28574054/kafka-consumer-rebalancing-algorithm/28580363#28580363 – serejja Sep 12 '15 at 16:56
-
@serejja Thank you, you could answer bellow – Neo Sep 14 '15 at 02:09
1 Answers
2
I'll give a bit more detailed answer here.
So the first point is that there is no reason to have more consumer threads (and each consumer has at least 1 consumer thread) than the number of partitions being consumed. The reason is that if you have more consumer threads than partitions, some consumer threads will just end up being idle and will just waste resources. So given the example you attached there is no point of having 4 consumers for 3 partitions.
The second point - the partition assignment depends on the strategy chosen by consumers in the group. Currently there are 2 partition assignment strategies - Range and RoundRobin. If you are using the Range strategy you can predict what partitions will be consumed by each consumer after rebalance. With RoundRobin strategy though you can't predict beforehand the partition assignments for consumers after rebalance.
The detailed answer that explains how consumer rebalancing works and how partitions are assigned is here.
You can also view current partition assignments for your consumer group in Zookeeper at /consumers/[group_id]/owners/[topic]/[partition]
