3

I have 2 different application instance consuming message from a topic, both application have same values for group.id and consumer.id

Will message is read by only 1 application or by both application? Also if we have same consumer.id in 2 different application, will it considered as 2 consumers in one group or only single consumer in that group?

E.g. App1 instance, group.id = conGrp1, consumer.id = consumer
App2 instance, group.id = conGrp1, consumer.id = consumer1

Do we still have only one group with one consumer, even though 2 different application instance are running?

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Atul Kumar
  • 31
  • 1
  • 6

1 Answers1

1

both application have same values for group.id and consumer.id

Then, both are part of the same group, each consumer not overlapping in consumed data - any given message is only seen in one consumer of the group

The group id is what determines this behavior. The consumer id is just a friendly name to find in the metrics or consumer group command, I believe, so you still only have one group. I'm not it's a good idea to put unique instances of the application with the same consumer id, even if part of the same group

difference between groupid and consumerid in Kafka consumer

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • I understand about group behavior. But still have doubt on consumer Id. So my question is if we have same consumer Id in two app instance, does group have 2 consumer or 1 consumer. See I am using 2 app instance for fault tolerance, if one app crashes , other will consume messages. – Atul Kumar Dec 05 '18 at 16:19
  • Well, `consumer.id` is not on the documentation anymore, so I don't know what it really is useful for. http://kafka.apache.org/documentation/#consumerconfigs. That being said, `group.id` is the only thing taken into account here – OneCricketeer Dec 05 '18 at 23:33
  • There are two processes running and polling from the topic, therefore you have two consumers in a group. When you describe the group, then you might see only one `consumer.id`, however I am under the assumption the value is automatically set, so why are you setting it? – OneCricketeer Dec 05 '18 at 23:47
  • ok...so how we are going to set number of consumers with in a group? – Atul Kumar Dec 06 '18 at 05:17
  • Will the consumer count now depends directly on topic partitions? E.g. if we have 2 topic in same group .i.e Group1. Topic1 has 2 partition and Topic2 has 3 partition, then does Kafka Consumer will automatically defines 5 consumers with in Group1? – Atul Kumar Dec 06 '18 at 05:51
  • No, it does not. You don't set that, because it just happens when you start multiple application with the same group id that they all rebalance and join a group – OneCricketeer Dec 06 '18 at 14:04