3

I'm testing kafka's partition reassignment as a precursor to launching a production system. I have several topics with 9 partitions each and a replication factor of 3. I've killed one of the brokers to simulate a failure condition and verified that some topics became under replicated (verification done via a fork of yahoo's kafka manager modified to allow adding a version 0.10.0.1 cluster).

I then started a new broker with a different id. I would now like to distribute partitions to this new broker. I attempted to use kafka manager's reassign partitions functionality however that did not work (possibly due to an improperly modified fork).

I saw that kafka comes with a bin/kafka-reassign-partitions.sh script but the docs say that I have to manually write out the partition reassignments for each topic in json format. Is there a way to handle this without manually deciding on which brokers partitions must go?

novon
  • 973
  • 3
  • 15
  • 30

1 Answers1

4

Hmm what a coincidence that I was doing exactly the same thing today. I don't have an answer you're probably going to like but I achieved what I wanted in the end.

Ultimately, what I did was executed the kafka-reassign-partitions command with what the same tool proposed for a reassignment. But whatever it generated I just replaced the new broker id with the old failed broker id. For some reason the generated json moved everything around.

This will fail (or rather never complete) because the old broker has passed on. I then had to delete the reassignment operation in zookeeper (znode: admin/reassign_partitions or something).

Then I restarted kafka on the new broker and it magically picked up as leader of the partition that was looking for a new replacement leader.

I'll let you know if everything is still working tomorrow and if I still have a job ;-)

ekeyser
  • 597
  • 4
  • 9
  • You mean you used yahoo's kafka manager to generate the reassignment json? – novon Sep 21 '16 at 03:23
  • I used the cli utils - kafka-reassign-partitions (or kafka-reassign-partitions.sh). I installed kafka from homebrew but kafka installations include the utils as well. – ekeyser Sep 21 '16 at 17:59
  • I also used Netflix's Exhibitor package for Zookeeper which allows you to view/modify zookeeper znodes. But you can manage Zookeeper via its utils. – ekeyser Sep 21 '16 at 18:00
  • Yep I'm using exhibitor as well. I didn't realize the cli tool actually generated the reassignment json, I'll need to read the docs again. – novon Sep 22 '16 at 01:51
  • kafka-reassign-partitions --generate --zookeeper zookeeper1:2181/znode --broker-list "id1,id2,id3..." --topics-to-move-json-file somefile.json – ekeyser Sep 22 '16 at 20:21
  • @ekeyser So how is everything going on now? ^_* – srain Dec 04 '18 at 09:30
  • Hah. I left that job although when I was still there I tried to float the idea that Kafka was too complex and unreliable for what we were doing and to use something else. Alas, it would've generated a trivial amount of work for the team and hence no traction was gained ;-) – ekeyser Jan 03 '19 at 21:24