0

Suppose I have a Kafka cluster of 1000 nodes. Let's say that some producer sends produces a message to certain topic, what is the first node in the cluster that receives this message? Is there any load balancer inside the Kafka cluster? Let's say that this node doesn't have the lead partition that the message should be pushed to, can it forward the message to the broker that includes this partition?

CrazySynthax
  • 13,662
  • 34
  • 99
  • 183

1 Answers1

1

The client knows all brokers of the cluster, and can only produce to the leader topic partition, which it can identify, and is only hosted on a single broker. The Controller defines leadership How many Kafka controllers are there in a cluster and what is the purpose of a controller?

There's no load balancer, all brokers are equal with respect to what requests they can receive and don't redirect events to others outside of replication

The partitioner instance within the prouder client will hash the key of the message (by default, and if key is non null), and modulo by the number of partitions in the topic. With that value, it'll produce to the specific broker hosting that partition number

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • How does the client knows all broker of the cluster? Does the client talk to one of the brokers which gives it the list of all other brokers? How does the client know to which broker to call? – CrazySynthax Aug 14 '21 at 18:37
  • 1
    That's the responsibility of the controller, but also bootstrap process (you can give the client one address, then all are returned from communicating with the controller). Overall, there's far more than one "single request" before a record is ever sent by your app. Are you overall concerned about the network traffic, for some reason? – OneCricketeer Aug 14 '21 at 19:11