changing kafka retention period during runtime

Question

With Kafka 0.8.1.1, how do I change the log retention time while it's running? The documentation says the property is log.retention.hours, but trying to change it using kafka-topics.sh returns this error

$ bin/kafka-topics.sh --zookeeper zk.yoursite.com --alter --topic as-access --config topic.log.retention.hours=24
Error while executing topic command requirement failed: Unknown configuration "topic.log.retention.hours".
java.lang.IllegalArgumentException: requirement failed: Unknown configuration "topic.log.retention.hours".
    at scala.Predef$.require(Predef.scala:145)
    at kafka.log.LogConfig$$anonfun$validateNames$1.apply(LogConfig.scala:138)
    at kafka.log.LogConfig$$anonfun$validateNames$1.apply(LogConfig.scala:137)
    at scala.collection.Iterator$class.foreach(Iterator.scala:631)
    at scala.collection.JavaConversions$JEnumerationWrapper.foreach(JavaConversions.scala:479)
    at kafka.log.LogConfig$.validateNames(LogConfig.scala:137)
    at kafka.log.LogConfig$.validate(LogConfig.scala:145)
    at kafka.admin.TopicCommand$.parseTopicConfigsToBeAdded(TopicCommand.scala:171)
    at kafka.admin.TopicCommand$$anonfun$alterTopic$1.apply(TopicCommand.scala:95)
    at kafka.admin.TopicCommand$$anonfun$alterTopic$1.apply(TopicCommand.scala:93)
    at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:57)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:43)
    at kafka.admin.TopicCommand$.alterTopic(TopicCommand.scala:93)
    at kafka.admin.TopicCommand$.main(TopicCommand.scala:52)
    at kafka.admin.TopicCommand.main(TopicCommand.scala)

I got a stupid question...is a kafka message part of a log? In other words, setting the retention for a log means setting a retention for these messages. The kafka docs don't exactly make that clear to me. — dtc, Jan 21 '21 at 21:59
kafka messages are stored in log segments, so I think the answer is yes — Foo L, Jan 22 '21 at 19:24

score 161 · Accepted Answer · edited Aug 24 '20 at 21:59

161

log.retention.hours is a property of a broker which is used as a default value when a topic is created. When you change configurations of currently running topic using kafka-topics.sh, you should specify a topic-level property.

A topic-level property for log retention time is retention.ms.

From Topic-level configuration in Kafka 0.8.1 documentation:

Property: retention.ms
Default: 7 days
Server Default Property: log.retention.minutes
Description: This configuration controls the maximum time we will retain a log before we will discard old log segments to free up space if we are using the "delete" retention policy. This represents an SLA on how soon consumers must read their data.

So the correct command depends on the version. Up to 0.8.2 (although docs still show its use up to 0.10.1) use kafka-topics.sh --alter and after 0.10.2 (or perhaps from 0.9.0 going forward) use kafka-configs.sh --alter

$ bin/kafka-topics.sh --zookeeper zk.yoursite.com --alter --topic as-access --config retention.ms=86400000

You can check whether the configuration is properly applied with the following command.

$ bin/kafka-topics.sh --describe --zookeeper zk.yoursite.com --topic as-access

Then you will see something like below.

Topic:as-access  PartitionCount:3  ReplicationFactor:3  Configs:retention.ms=86400000

edited Aug 24 '20 at 21:59

Ricardo

3,696
5
36
50

answered Mar 19 '15 at 04:06

Heejin

4,463
3
26
30

I did the above command, but it doesn't work! I can still can see the old logs. Any ideas please. – Jack Sep 13 '15 at 23:07
1

@Jack retention is a guarantee on minimum time span to retain data. If you see older logs that is retaining for at least the minimum. There are other configuration options for a cleanup interval and retention size that might help delete older logs. – dlamblin Oct 18 '15 at 14:46
2

Actually, this retention might never work. Because whenever it is trying to apply the retention policy the log files may be in use or the writer stream is kept open. That's why it always fails and the console/app log file reports this actually - check this out https://issues.apache.org/jira/browse/KAFKA-1194 – ha9u63a7 Jun 05 '17 at 09:20
45

Note that `kafka-topics.sh` is now deprecated. The new way to do that is: `$ bin/kafka-configs.sh --zookeeper zk.yoursite.com --alter --entity-type topics --entity-name mytopic --add-config 'retention.ms=86400000'` – Romain Hardouin Oct 09 '17 at 08:05
Does anyone know if the offsets for each topics are also controlled by this "retention.ms" property? – user3366706 Nov 21 '17 at 19:52
4

And once you are done with your testing, you can revert the retention time back to default value using: `$ bin/kafka-configs.sh --zookeeper :2181 --entity-type topics --entity-name --alter --delete-config retention.ms` – Sandeep Kanabar Jan 15 '18 at 17:57
1

@user3366706 no, the offsets are controlled by a different set of configs. At times, this can be confusing because if your offsets are only stored for 24 hours but your data is stored for 7 days, if you don't set new offsets for 25 hours (app is down or something) when you restart the app it will read from the beginning or end instead of the last offset. It's usually suggested you keep the retention.ms and offset retention the same or close. – ammills01 Apr 26 '18 at 10:35
@ammills01 yeah looks like "offsets.retention.minutes" controls offset retention time, but as per documentation it is at the broker level, not topic level. – user3366706 Apr 27 '18 at 20:24
@RomainHardouin anyone know if i have 4 topic and i want to configure all four combine with retention.byte of 5 GB. How can i configure same? – ankit Jul 25 '19 at 10:15
what is the behavior when topic level (updated) retention value exceeds the server level default? (say broker level is 7 days but you set the topic level value to 15 days, which value would apply to that topic?) – Sachin Sharma Sep 06 '21 at 07:07

score 70 · Answer 2 · answered Jun 21 '17 at 18:32

70

The following is the right way to alter topic config as of Kafka 0.10.2.0:

bin/kafka-configs.sh --zookeeper <zk_host> --alter --entity-type topics --entity-name test_topic --add-config retention.ms=86400000

Topic config alter operations have been deprecated for bin/kafka-topics.sh.

WARNING: Altering topic configuration from this script has been deprecated and may be removed in future releases.
     Going forward, please use kafka-configs.sh for this functionality`

answered Jun 21 '17 at 18:32

Vikas Tikoo

1,545
11
14

if i have 4 topic and i want to configure all four combine with retention.byte of 5 GB. How can i configure same? – ankit Jul 25 '19 at 10:17
@ankit I don't think its possible to provide a single retention config for a group of topics. – Vikas Tikoo Jul 26 '19 at 18:01

Foo L · Answer 3 · 2015-03-19T06:48:03.060

8

The correct config key is retention.ms

$ bin/kafka-topics.sh --zookeeper zk.prod.yoursite.com --alter --topic as-access --config retention.ms=86400000
Updated config for topic "my-topic".

edited Mar 19 '15 at 06:48

answered Mar 18 '15 at 17:50

Foo L

10,977
8
40
52

1

if i have 4 topic and i want to configure all four combine with retention.byte of 5 GB. How can i configure same? – ankit Jul 25 '19 at 10:16

score 4 · Answer 4 · edited Jun 05 '20 at 21:29

4

I tested and used this command in kafka confluent V4.0.0 and apache kafka V 1.0.0 and 1.0.1

/opt/kafka/confluent-4.0.0/bin/kafka-configs --zookeeper XX.XX.XX.XX:2181 --entity-type topics --entity-name test --alter --add-config  retention.ms=55000

test is the topic name.

I think it works well in other versions too

edited Jun 05 '20 at 21:29

Foo L

10,977
8
40
52

answered May 22 '18 at 17:01

Alihossein shahabi

4,034
2
33
53

score -1 · Answer 5 · answered Jun 13 '23 at 19:19

-1

--zookeeper is depricated for kafka-configs.sh, instead use:

kafka-configs.sh --bootstrap-server <yourserver:9092> --alter --topic <your-topic> --add-config <config-name=value>

answered Jun 13 '23 at 19:19

nCoder

88
1
13

changing kafka retention period during runtime

5 Answers5

Linked