1

We are seeing a steady growth in the persistent storage for Kafka, around 10% per day - effectively implying the configs are not tuned. There are 3 brokers and 5 topics. Retention was configured for 2 hours.

Following are the config parameters set -

Topic level :

   retention.ms=7200000
   delete.retention.ms=7200000
   segment.ms=7200000
   cleanup.policy=delete
   file.delete.delay.ms=1
   segment.bytes=150000

Broker Level:

"log.cleaner.delete.retention.ms": "7200000"
"log.retention.ms": "7200000"

Any help would be appreciated. Thanks in advance.

prof
  • 21
  • 3
  • Do you have access to the Kafka logs to check if the cleaner is running? https://jaceklaskowski.gitbooks.io/apache-kafka/content/kafka-log-LogCleaner.html – Gerard Garcia Aug 04 '22 at 08:21
  • 1
    Kafka brokers are always going to keep at least one segment open per partition. Would it be possible that some partition are receiving so little traffic that they keep having one single partition < 15Mb that keeps growing, either due to traffic imbalance or too high number of partitions per topic? – Svend Aug 04 '22 at 08:53

1 Answers1

1

Seems to have fixed this issue.. Steps performed -

  1. logged into kafka broker
  2. fetch the list of files in the descending order of their sizes from the /opt/kafka/data-0/logs/ path.
  3. mapped the file to a topic already present in kafka - should be easily identifiable because of the naming convention
  4. applied the above settings on each of the topics - whose log files were greater than 10 MB
  5. retention time set to 30 minutes, instead of 3 hours as posted in the question.
prof
  • 21
  • 3