How do I ensure that logs are retained forever in Kafka?

Question

I need to configure Kafka never to delete logs. Looking at their documentation, I see two kinds of parameters that control this:

• The "log cleaner", described at http://kafka.apache.org/documentation.html#compaction. It seems that setting log.cleaner.enable = false prevents the cleaner from deleting or compacting logs.

• The "log retention" mechanism, described at http://kafka.apache.org/documentation.html#configuration under log.retention.<time scales>. It seems that setting log.retention.ms = -1 signals infinite retention.

My question is:

What is the relationship between these two mechanisms, are these the only two, and are the above settings necessary and sufficient to prevent logs from ever being deleted?

Answer 1

Not sure which version you are using. For me, I use this:

log.retention.hours=2147483647

That's roughly 245,000 years. I believe I tried -1 at some point, but using the max value worked. Perhaps log.retention.ms accepts different values, but regardless -- the second of your configuration options should be sufficient. My guess is neither option is necessary; either would be sufficient. Certainly, the second one is sufficient by itself.

Answer 2

As said in this answer, you can set -1 for log.retention.hours and log.retention.bytes.