Kafka streams is creating a reparation topic in the kafka cluster with Retention.ms=-1, I think because of this reason I am unable to consume the date from the topic.
Why it is creating with -1ms? How to let it create with a custom ms?
Asked
Active
Viewed 94 times
1 Answers
0
Fact
retention.ms for the repartition topics is -1 by default and there's no way to override this value in kafka-streams client code. What I misunderstood
Size of the repartition topic would be increasing infinitely since the retentions.ms for the repartition topics is -1. Fix misunderstanding
There's a method called maybeCommit in the StreamThread class. maybeCommit method is called iteratively inside the loop that handles stream records. Inside the maybeCommit method (version 2.7.1), there's a comment like below. try to purge the committed records for repartition topics if possible
https://github.com/apache/kafka/blob/2.7.1/streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamThread.java#L923-L926 Based on this, what I understand is that when the record in the repartition topics is streamed down to the changelog topic, then the records already sent are purged periodically. Therefore, there's no need to clear or manage retention.ms for the repartition topics. Reference
https://issues.apache.org/jira/browse/KAFKA-6150 Please leave a comment or correct this if I'm wrong.
Krissag
- 17
- 4