24

I am writing log messages into a Kafka Topic and I want the retention of this topic to be permanent. I have seen in Kafka and Kafka Connect (_schemas, connect-configs, connect-status, connect-offsets, etc) that there are special topics that are not deleted by the log retention time. How do I enforce a topic to be like these other special topics? Is it the naming convention or some other properties?

Thanks

user1077071
  • 901
  • 6
  • 16
  • 29

3 Answers3

45

If you want to retain all topics forever, you can set both log.retention.hours and log.retention.bytes to -1.

highlycaffeinated
  • 19,729
  • 9
  • 60
  • 91
  • 8
    It appears that prior to April 2015 the way to do this was to set `log.retention.hours` to `2147483647`. Then a [JIRA](https://issues.apache.org/jira/browse/KAFKA-1990) added the -1 feature. That JIRA, however, did not explicitly state what value is interpreted as "retain forever". I dug around the code a little and indeed, it's [-1](https://github.com/apache/kafka/blob/0.11.0.2/core/src/main/scala/kafka/server/KafkaConfig.scala#L1165). You can also set -1 for [hours, minutes or ms](https://github.com/apache/kafka/blob/0.11.0.2/core/src/main/scala/kafka/server/KafkaConfig.scala#L1078-L1090) – Dmitry Minkovsky Dec 02 '17 at 21:03
  • 1
    Is there any reason why we shouldn't do this other than too much space used? Like is there a scalability or speed issue as apart of this? – cody.tv.weber Feb 05 '19 at 14:31
  • 1
    something worth noting is the limitation is space per partition, and each partition is limited to the size of a disk. indirectly, that may impact scalability but other than that I don't think there is anything. – Matt Howlett Apr 14 '19 at 23:02
  • Are you sure log.retention.hours work? Context: https://stackoverflow.com/a/70535714/4106031 – Alok Kumar Singh Dec 31 '21 at 02:59
7

These special topics are compacted topics. This means they are made up of keyed messages and only keep the list recent key. A full write here. This is probably what you want for infinite retention time in most cases.

dawsaw
  • 2,283
  • 13
  • 10
  • How to retain data for only one topic(keep the data forever) ? log.retention.bytes and log.retention.ms are applicable to all topics, but not to only a single topic. – VinayKumar Jun 01 '21 at 12:35
  • Example of per-topic config at https://www.baeldung.com/kafka-message-retention – Ryan Apr 27 '22 at 06:34
1

You can find the default values for parameters here: https://jaceklaskowski.gitbooks.io/apache-kafka/content/kafka-properties.html

log.retention.bytes and log.retention.ms can be set to -1 if you want to keep the data in topics forever.

lordzuko
  • 773
  • 10
  • 22