4

I performed reassignment of a topic within my cluster. There were some problems throughout (running out of disk space on one of the target brokers), but I managed to fix it and the process completed succesfully.

It seems, however, that while one of the partitions was reassigned to another brokers, the data was not removed from the source broker's disk. And since the partion is quite big, I'd like it gone.

For obvious reasons I do not want to login to shell and rm -rf the directory. What would be the steps I could take to debug why the data was not deleted and then how to "encourage" the cluster to performe a clean up?

For a while thought that the retention policy might kick in and delete the data, but it's set to run every 10 minutes and it's been over a day since the reassignment has finished.

The replicas are as follows:

# bin/kafka-topics.sh -describe --zookeeper 1.2.3.4 --topic topic-name
Topic:topic-name    PartitionCount:4        ReplicationFactor:3     Configs:retention.ms=3153600000000,compression.type=lz4
        Topic: topic-name   Partition: 0    Leader: 1014    Replicas: 1014,1012,1002        Isr: 1012,1002,1014
        Topic: topic-name   Partition: 1    Leader: 1007    Replicas: 1007,1006,1003        Isr: 1006,1007,1003 <--- this is the partition
        Topic: topic-name   Partition: 2    Leader: 1013    Replicas: 1013,1008,1001        Isr: 1013,1008,1001
        Topic: topic-name   Partition: 3    Leader: 1011    Replicas: 1011,1016,1010        Isr: 1010,1011,1016

And here we can see that the broker 1008 holds two partitions: 2 (it should) and 1 (it should not, this we need gone).

/data_disk_0/kafka-logs# cat meta.properties | grep broker.id
broker.id=1008

/data_disk_0/kafka-logs# du -h --max-depth=1 . | grep topic-name-1
295G    ./topic-name-2
292G    ./topic-name-1

edit: What's curious, all files in the topic directory (/data_disk_0/kafka-logs/topic-name-1/*) are opened by Kafka (checked with lsof). I don't know whether it's a default behaviour for Kafka to read all files in its data dir regardless of their status or it means that these files are still being somehow used.

theo
  • 155
  • 2
  • 13
  • You just want to delete the partition, right? – Nijat Mursali Sep 05 '19 at 10:29
  • Yes, that's right. – theo Sep 05 '19 at 10:50
  • I actually think that rm -rf would not do any damage. Take a look here https://stackoverflow.com/questions/33537950/how-to-delete-a-topic-in-apache-kafka . Since you need just to empty one of your brokers I think your can remove topic data just from that broker. Partition you are trying to delete is not a leader and not even in list of replicas. You can take a look at messages in that partition, to see if that partition replica is receiving new messages. Please, write here your conclusions after you find out problem, I would also like to know what was the problem. – Spasoje Petronijević Sep 05 '19 at 13:59
  • Don't know the exact source of the problem, but I believe it has something to do with running out of disk space during the original reasignment. Anyway - Kafka was keeping the files open, but no data was written to the partition in a few days, so I decided to stop broker and manually delete the data. So far no problems observed. – theo Sep 10 '19 at 06:53

1 Answers1

1

It is not possible to delete partitions from a topic in Kafka. A partition is a file, Kafka assigns data to each partition depending on the Key. Let's say the partition 2 has data for the Key AAA, but the AAA Key isn't longer produced then you might see partition 2 not used.

Take a look to this video: https://developer.confluent.io/learn-kafka/apache-kafka/partitions/#:~:text=Kafka%20Partitioning&text=Partitioning%20takes%20the%20single%20topic,many%20nodes%20in%20the%20cluster.

The only way is to delete the topic and create it again with the correct number of partitions

Luis Estrada
  • 371
  • 7
  • 20