7

I am using kafka_2.10-0.10.0.1. I created the topics with 1 partitions. I know that the default cleanup policy is "delete". I want to keep all the records in the topic all the time (without deleting any record). What's the right way : make "log.cleaner.enable=false" or "log.cleanup.policy=compact"?

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
DaliMidou
  • 111
  • 1
  • 3
  • 14

1 Answers1

7

Topics have broker-wide configs that apply by default to any topic that doesn't have a config, but topics can also have topic-specific configs that override or complement broker-wide topic configs.

Broker-wide topic configs are set in your service.properties file. Topic specific configs are set using the bin/kafka-topics.sh script or AdminClient if you are using Java.

The relevant broker-wide config for you is log.retention.ms and the equivalent topic-specific config is retention.ms. If you set log.retention.ms to -1, all topics without this the retention.ms config will have an infinite retention period. Likewise, if you set -1 for retention.ms on a specific topic, it will have an infinite retention period.

To set retention.ms on a new topic:

bin/kafka-topics.sh --zookeeper zk_host:port/chroot --create --topic my_topic_name
  --partitions 20 --replication-factor 3 --config retention.ms=-1

You can also modify an existing topic to set retention.ms to -1:

> bin/kafka-configs.sh --zookeeper zk_host:port/chroot --entity-type topics --entity-name my_topic_name --alter --add-config retention.ms=-1

See the full list of topic-specific configs here: https://kafka.apache.org/documentation/#topicconfigs and more about topic operations: https://kafka.apache.org/documentation/#basic_ops

softarn
  • 5,327
  • 3
  • 40
  • 54
Dmitry Minkovsky
  • 36,185
  • 26
  • 116
  • 160
  • Given this description: "This configuration controls the maximum time we will retain a log before we will discard old log segments to free up space if we are using the "delete" retention policy. This represents an SLA on how soon consumers must read their data. If set to -1, no time limit is applied." I think compaction will still happen, even if this is set to -1 if the cleanup policy is "compact". I've just had this config on our setup and it compacted old logs. – Rambatino Jun 08 '22 at 13:38