Can we have retention period of zero in Kafka broker?

Question

Does retention period of zero makes sense in kafka borker?

We want to quickly forward message from producer to consumer via kafka broker. From buffercache/pagecache on broker machine without flushing to disk. We do not need replication and assume our broker will never crash.

Do you mean that you want to keep the messages forever or to delete them immediately after consumption? — Giorgos Myrianthous, Mar 15 '19 at 20:21
@GiorgosMyrianthous delete immediately after consumption and preferably without writing to disk...producer to broker pagecache to consumer..can it be done? or it needs to be flushed first to OS by broker before consumer can poll it — Youli Luo, Mar 15 '19 at 20:25
This might be helpful: https://stackoverflow.com/questions/54046617/kafka-retention-period-parameter/54049085#54049085 — Bitswazsky, Mar 17 '19 at 04:40

score 5 · Accepted Answer · answered Mar 15 '19 at 20:39

When a message is produced to a Kafka topic it is written to the disk. Once the message has been consumed, the offset of this message is committed by the consumer (if you are using the high-level consumer API) however, there is no functionality that deletes only the messages that have been consumed (many consumers may subscribe to the same topic and some of them might have consumed that message while some others might have not).

What I would suggest in your case is to set a short retention period (which by default is set to 7 days) but allow a reasonable amount of time in order to allow your consumer to consume the messages. To do this, you simply need to configure the following parameter in server.properties:

log.retention.ms=X

Note that there is no guarantee that the deleted message(s) have been successfully consumed by your consumer(s). For example, if you set the retention period to 2 seconds (i.e. log.retention.ms=2000) and your consumer crashes, then every message which is sent to the topic while the consumer is down will be lost.

Can we have retention period of zero in Kafka broker?

1 Answers1