3

We have 6 node kafka cluster. Out of the 6 machines, 3 of them have both Kafka + zookeeper, and the remaining 3 of them have just kafka. Recently, we added one more kafka node. While re-assigning the partition to all the nodes (including the newer one), we executed the below command:

/opt/kafka/bin/kafka-reassign-partitions.sh --reassignment-json-file new_assignment_details.json --execute --zookeeper localhost:2181

However, when we verified the status by using the below command,

/opt/kafka/bin/kafka-reassign-partitions.sh --reassignment-json-file new_assignment_details.json --verify --zookeeper localhost:2181

We get the below output. Some of the partitions are re-assignment is still in progress.

/opt/kafka/bin/kafka-reassign-partitions.sh --reassignment-json-file new_assignment_details.json --verify --zookeeper localhost:2181 | grep 'progress'
Reassignment of partition [topic-name,854] is still in progress
Reassignment of partition [topic-name,674] is still in progress
Reassignment of partition [topic-name,944] is still in progress
Reassignment of partition [topic-name,404] is still in progress
Reassignment of partition [topic-name,314] is still in progress
Reassignment of partition [topic-name,853] is still in progress
Reassignment of partition [prom-metrics,403] is still in progress
Reassignment of partition [prom-metrics,134] is still in progress

There is no way to either:

  1. Cancel the on-going partition re-assignment.
  2. Rollback is also not possible. ( When we try doing that it says that "There is an existing assignment running."

Kafka Version: 0.10.1.1

Platform: Amazon Linux 1 x86_64 GNU/Linux

Can anyone please help me out in this?

Vikash Kumar
  • 177
  • 1
  • 9

3 Answers3

2

You can cancel the on-going partition re-assignment by deleting the zk node: delete /admin/reassign_partitions. Then you can rollback or reassign with another json file.

itissonan
  • 21
  • 2
1

In my case, for canceling current partition re-assignment and being able to perform another re-assignment you should:

  1. Delete /admin/reassign_partitions zk node
  2. Delete /controller zk node to re-elect the active controller

UPDATE
consider using --additional argument of kafka-reassign-partitions.sh!

Mostafa Bayat
  • 93
  • 6
0

Following on @Mostafa-Bayat, version 0.10 is very old and administration is manual and error prone. The AdminAPI was included in kafka 2.6 (I believe) and improved in following version. (see KIP-236)

Everyone should be on the latest 2.x (2.8.1 at time of writing) or latest 3.x (3.2.1 currently) to take advantage of the adminapi that makes reassignments possible with kafka-reassign-partitions.sh without manual edits in zookeeper. (3.x is working to remove zk entirely).

Phil
  • 1,226
  • 10
  • 20