1

on all our kafka machines ( production machines ) , we see that: ( no free space )

df -h /var/kafka 
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb         11T   11T  2.3M 100% /var/kafka

and under /var/kafka/kafka-logs

we see all topic folders (huge size) as example:

117G hgpo.llo.prmt.processed-28 
117G hgpo.llo.prmt.processed-29 
117G hgpo.llo.prmt.processed-3 
117G hgpo.llo.prmt.processed-30 
117G hgpo.llo.prmt.processed-31 
117G hgpo.llo.prmt.processed-32

what is the best approach to delete the topic/s from the folder /var/kafka/kafka-logs ,

and what are the exactly steps to do so , as stop service before deletion etc .

second important question:

what is the mechanizem that suppose to delete automatically the topics ?

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
King David
  • 500
  • 1
  • 7
  • 20
  • What did you set the topic retention period to? How large are your messages? – OneCricketeer Nov 30 '17 at 19:14
  • actually I not set it , do we need to set it ? if yes then to which value and how ? – King David Nov 30 '17 at 19:45
  • Topics are retained for a week by default. https://kafka.apache.org/documentation/#topicconfigs The fact that you have 30+ partitions that are 100+GB, sounds like your topic is either compacted, or the retention period is way too high. Kafka isn't intended to be an indefinitely persistent datastore for such large datasets – OneCricketeer Nov 30 '17 at 19:54
  • yes but the partition now is 100% and we not have any other choice , we must to delete some of the topics to reduce the partition usage – King David Nov 30 '17 at 20:02
  • https://stackoverflow.com/a/25166222/2308683 – OneCricketeer Nov 30 '17 at 20:13
  • Possible duplicate of [Purge Kafka Topic](https://stackoverflow.com/questions/16284399/purge-kafka-topic) – OneCricketeer Nov 30 '17 at 20:25
  • just one note , I see that we need to set it - kafka-topics.sh --zookeeper localhost:13003 --alter --topic MyTopic --config retention.ms=1000 before topic deletion - bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic test , can you approve this ? – King David Nov 30 '17 at 20:34
  • There is no reason to alter a topic, if you are just going to delete it anyway – OneCricketeer Nov 30 '17 at 20:38
  • Also, not sure where you got `--zookeeper localhost:13003`, but Zookeeper typically is on `2181` – OneCricketeer Nov 30 '17 at 20:39

0 Answers0