0

I come across this phrase from https://niqdev.github.io/devops/kafka/ and https://livebook.manning.com/book/kafka-streams-in-action/chapter-2/109 (Kafka Streams in Action )

The controller broker is responsible for setting up leader/follower relationships for all partitions of a topic. If a Kafka node dies or is unresponsive (to ZooKeeper heartbeats), all of its assigned partitions (both leader and follower) are reassigned by the controller broker.

I think it is not correct assignment of follower partitions to other brokers - as the partitions wont heal themselves unless the broker comes back . I know it ONLY happens for leader replica where if the broker that has leader replica gone down, one of the broker that contains follower will become leader. But, I dont think "reassigment" of followers will happen automatically unless reassignment is initiated manually. Please add your inputs

Nag
  • 1,818
  • 3
  • 24
  • 41
  • Does this answer your question? [How many Kafka controllers are there in a cluster and what is the purpose of a controller?](https://stackoverflow.com/questions/49525141/how-many-kafka-controllers-are-there-in-a-cluster-and-what-is-the-purpose-of-a-c) – LimorAbv Aug 08 '20 at 13:30

1 Answers1

1

The terminology might be a little off indeed but still applies. Followers are not necessarily assigned to other brokers but they need to change the endpoint to where they are going to send fetch requests. The follower's job is to stay in-sync with the leader, and if the leader has been assigned to a new broker because the old one failed then the followers need to send their fetch requests to the new elected broker. I think that is what reassignment means in the context that you shared.

Ricardo Ferreira
  • 1,236
  • 3
  • 6
  • Please see this phrase "all of its assigned partitions (both leader and follower) are reassigned by the controller broker." - this phrase specifically says about followers being reassigned to other brokers . I guess this is not correct . However, leaders will be reassigned as in a follower which is ISR will be chosen as leader. Please confirm – Nag Jul 24 '20 at 03:29
  • "leaders will be reassigned as in a follower which is ISR" -- Yes, and that is essentially what I have written in the answer provided. – Ricardo Ferreira Jul 24 '20 at 13:11
  • i got it. But, is it true for "followers" as well ? I mean , if one of the brokers went offline, will the "followers" on that broker will be reassigned to other brokers ? The reason why i am asking this question was , I didn't see this behavior happening – Nag Jul 24 '20 at 13:46
  • No, followers won't be moved since this would be an expensive operation. It is smarter to just keep them where they are and making them aware that a new leader was selected so they can continue to keep in-sync with the new data stream. I hope that clarifies. – Ricardo Ferreira Jul 24 '20 at 18:49
  • Yes, I was under the same impression. That is the reason I asked for clarification as this statement contradicts that. Please see the phrase in quotes "followers" and it is giving impression as if the followers will be moved(reassigned) which is not the case. - If a Kafka node dies or is unresponsive (to ZooKeeper heartbeats), all of its assigned partitions (both leader and "follower") are reassigned by the controller broker. – Nag Jul 25 '20 at 05:12