End to end procedure in order to remove kafka topics

Question

We have ambari cluster with HDP version 2.6.4 , while kafka version 0.10.0.2.6 ( vendor - hortonworks , OS - redhat 7.2 version )

We have 3 kafka brokers and all topics partitions are under /var/kafka/kafka_data folder

Kafka brokers ids are:

./zookeeper-shell.sh zookeper01:2181 <<< "ls /brokers/ids"
Connecting to zookeper01:2181
Welcome to ZooKeeper!
JLine support is disabled

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
[103, 102, 101]

We decided in this stage after kafka cluster works more then 2 years to delete all topics from all kafka brokers

Example of our topics from kafka machine

[root@kafka01 ~]# ./kafka-topics.sh --zookeeper zookeper01:2181 --list
__consumer_offsets
_schemas
ambari_kafka_service_check
mk.term.control.ova
mk.term.control.ovb
mk.term.control.ovc
mk.term.control.ovc
mk.sum.control.ova
mk.sum.control.ovb
mk.sum.control.ovc
mk.sum.control.ovc
mk.sum.control.ovd
mk.sum.control.ove
mk.sum.control.ovf
mk.sum.control.ovg
mk.sum.control.ova
mk.sum.control.ovb
.
.
.
.

Additionally I understand from searching in Hortonworks site that need also to remove the topics from zookeeper cli ( https://community.hortonworks.com/answers/list.html )

Unfortunately we not found complete procedure that described step by step Exactly how to remove the topics with all additionally aspects ( as stop the kafka brokers before topics deletion .. etc )

I would appreciate to get complete procedure that fit kafka Version - 0.1.X

no , because I don't have agreement with hortonworks , its costs money — Judy, Jul 23 '19 at 15:35
Possible duplicate of [How to Delete a topic in apache kafka](https://stackoverflow.com/questions/33537950/how-to-delete-a-topic-in-apache-kafka) — OneCricketeer, Jul 23 '19 at 17:03
what about __consumer_offsets _schemas topics , can we also delete them? as the others? — King David, Jul 23 '19 at 18:11
I dont think its duplicate the post - "How to Delete a topic in apache kafka" isn't described deeply the deletion for example - "Delete the topic directory with rm -rf command" ,,, is its mean each partition topic under the folder - /var/kafka/kafka_data ?? or remove complete the folder - /var/kafka/kafka_data ?? on each kafka ? — King David, Jul 23 '19 at 18:18
@cricket_007 I think you are the best person that can answer on Judy question , Judy want to understand step by step the deletion and not in highlights as described on "How to Delete a topic in apache kafka" — King David, Jul 23 '19 at 18:24
@cricket_007 , we are waiting for your answer, please share you answer — King David, Jul 23 '19 at 19:48
@KingDavid I've edited the answer there. You would ssh to the stopped brokers and physically wipe the Kafka data with `rm -rf` — OneCricketeer, Jul 23 '19 at 20:14
what about the topics - __consumer_offsets _schemas , do we need to delete them also ? — King David, Jul 23 '19 at 21:02
@cricket_007 please advice about - about the topics - __consumer_offsets _schemas , do we need to delete them also ? — King David, Jul 23 '19 at 22:00
@KingDavid If you want a brand new, fresh Kafka instance, then yes, but you would likely need to create them back manually... You could also format the entire data directories for both Kafka and Zookeeper (assuming they are separate disks than the OS)... The question did ask to delete **all** topics, no? — OneCricketeer, Jul 23 '19 at 22:06
@cricket_007 ok now we understood , all these steps are manual as you know and thanks to you , but do you know some tool or scripts that know to do all deletion automatically ? — King David, Jul 23 '19 at 22:09
@KingDavid Does this help? https://stackoverflow.com/a/48620787/2308683 — OneCricketeer, Jul 23 '19 at 22:15
@cricket_007 regarding what you suggest to king David, as I understand this approach delete the data from the topic partition not the topic itself - am I wrong ? — Judy, Jul 24 '19 at 09:49

End to end procedure in order to remove kafka topics

0 Answers0