9

My disk space is running out, after syncing 2 millions message in one of my Kafka topic. I even deleted the topic but still the space is somehow seems consumed. Any help of link would be appreciated.

Nadeem
  • 194
  • 1
  • 1
  • 8

4 Answers4

5

Use the below command to know how much disk space consumed by each topic

du -sh /tmp/kafka-logs/*
Kamal Chandraprakash
  • 1,872
  • 18
  • 28
  • The above command gives result like this: [root@MyDev ~]# du -sh /tmp/kafka-logs/* 4.0K /tmp/kafka-logs/activity_topic-0 4.0K /tmp/kafka-logs/callback_topic-0 4.0K /tmp/kafka-logs/callback_topic-1 4.0K /tmp/kafka-logs/callback_topic-2 4.0K /tmp/kafka-logs/callback_topic-3 4.0K /tmp/kafka-logs/callback_topic-4 4.0K /tmp/kafka-logs/__consumer_offsets-0 4.0K /tmp/kafka-logs/__consumer_offsets-1 4.0K /tmp/kafka-logs/__consumer_offsets-10 – Nadeem Jun 06 '16 at 12:27
  • How you deleted the topic ? – Kamal Chandraprakash Jun 07 '16 at 06:41
1

Another way to determine the size of the topic to send metrics, such as graphite. Then the size can be found as follows: kafka.log.Log.<topic_name>-<partition_number>-Size.value

Andrey
  • 485
  • 3
  • 17
  • Hi Andrey, thanks for your comment. This is actually a problematic answer. We have to keep in mind that Kafka is a distributed servcie, and partitions are not guaranteed to be perfectly balanced. You will want to run Kamal's command on each host. – robert Oct 05 '18 at 18:18
  • Hi, robert, of course, it is necessary to collect metrics from each broker, and to get complete size of the topic, you need summarize metrics from each broker for all partitions on this brockers. – Andrey Mar 27 '19 at 21:41
1
  • To delete messages from the topic you shall first set retention time to e.g. 1000ms. This will purge all messages in 1 sec. (see the answer: The correct config key is retention.ms)
  • Delete the topic
  • General think through the retention time for all topics, as default is 7 days, what might be quite much, when you operate with high frequecies.
Seweryn Habdank-Wojewódzki
  • 2,037
  • 4
  • 30
  • 51
0

From server.properties

#A comma separated list of directories under which to store log files

log.dirs=/tmp/kafka-logs

du will work but make sure to check all log directories. In fact, any normal unix command that you like to check disk space will work.

Deleting topic does not guarantee deletion. It will put it in "to be deleted" state. If you have any producer/consumer still attached to the topic, it will not be removed. Try the following:

  1. Make sure all brokers that have topic replicas are running
  2. Make sure you allow topic deletion delete.topic.enable = true
  3. Check if topic was deleted bin/kafka-topics.sh --list --zookeeper zookeeper_host:2181
YaRiK
  • 698
  • 6
  • 13