0

I am reading Kafka documentation and trying to understand the working of it. This is regarding consumers. In brief, a topic is divided into number of partitions. There are number of consumer groups, each having number of consumer instances. Now, my question is, does each partition sends sends "same" message to each consumer groups, which in turn is given to specific consumer instance within the group?

If it is, how does Kafka ensures the message is processed only by one consumer?

Kindly guide me if I am missing something.

Matthias J. Sax
  • 59,682
  • 7
  • 117
  • 137
Chinmay
  • 73
  • 3
  • 8
  • Related: https://stackoverflow.com/q/35561110/1531971 and maybe https://stackoverflow.com/q/48448066/1531971 (and advanced reading: https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/) –  Dec 13 '18 at 19:32
  • "If it is, how does Kafka ensures the message is processed only by one consumer?" It does not ensue this -- why should it? – Matthias J. Sax Dec 14 '18 at 08:23

1 Answers1

0

Well to put it simply :

  1. we have topic divided into partitions.

  2. we have consumer that consume data from thoses topics.

  3. Consumers are part of consumer group by sharing the same group.id.

  4. From a topic every partitions is consumed by one consumer within a consumer groups.

Example : Topic "test" with 3 partitions. Consumer group A : with 3 consumers Consumer group B : with 2 consumers.

Ths two consumer groups A and B consumes data from the topic "test".

Within the group A every consumer (so 3) will consume one partition each whereas in group consumer B (with 2 consumer) , one consumer will read 2 partitions and the other will consume the last one. If we have a last consumer group with only one consumer inside, it will read all 3 partitions of that topic.

Hope that's help, let me know if you didn't understand.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Saïd Bouras
  • 286
  • 1
  • 4
  • Hi, one question. In your eg. consider a message (msg1) in partition 1. Since partition 1 can be read by one consumer in group A and one consumer in group B. How does Kafka ensures that only one consumer(might be from A or B) processes the msg1. – Chinmay Dec 14 '18 at 00:46
  • Or is it like both the consumers will process the msg1? – Chinmay Dec 14 '18 at 00:47
  • Yes it's exactly that, the message is consumed by both consumer. – Saïd Bouras Dec 14 '18 at 08:26
  • Great. Could you tell me why is it send to both consumer and not just one. Consider there are 10 consumer groups. Each group will get the message. Why is this required? What is the purpose behind it? – Chinmay Dec 14 '18 at 20:10
  • Apache Kafka at his main goal it's publish-subscribe messaging system. – Saïd Bouras Dec 14 '18 at 20:15
  • Think of an application subscribing to a topic to get products of retail company for example, this application need those products for auditing purposes, but other people's need too to know all the products, imagine the marketing team for doing campaigns or web site for doing online shopping. – Saïd Bouras Dec 14 '18 at 20:21
  • Got it. I read an eg like Uber uses Kafka to send the ride requests msg to drivers. Wouldn't this cause a problem if two or more consumers are processing the same message to allocate a driver to requester? How would this fit ? – Chinmay Dec 14 '18 at 20:28
  • It's different from a traditional queuing system, see it like a publish subscribe messaging system. When you subscribe to a newsletter, you have notifications but you're not the only one, other people subscribe too to get the news. – Saïd Bouras Dec 14 '18 at 20:31
  • I don't know how uber uses kafka, but send the article link if you can, but in my experience of using uber, when you want a ride it connect to all available drivers in the area and it's the first that's respond ok who get the ride so i'ts pretty normal that every drivers has subscribed the request_driver topic. – Saïd Bouras Dec 14 '18 at 20:45
  • Thanks Said Bouras. It was helpful. I am not finding the page where I read about Uber. But what you say makes sense. – Chinmay Dec 14 '18 at 21:25