0

I am having a use-case of kafka streams where I need to perform aggregate operation with past data that might be consumed even months earlier.

I wonder if it means that I need to be concerned about default retention period of internal topics e.g. XXX-REDUCE-STATE-STORE-changelog, XXX-AGGREGATE-STATE-STORE-repartition and explicitly change/set somehow?

If yes, is there a way to configure it for stream app? If I set default retention period at broker level, will my newly created internal topics have forever retention?

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Vishal Shukla
  • 2,848
  • 2
  • 17
  • 19
  • Seems to be similar question https://stackoverflow.com/questions/65971984/why-retention-ms-of-kaka-streams-repartition-topic-is-set-to-1-by-default-isn – OneCricketeer Jan 31 '21 at 16:06
  • I think just the very final part of my question is similar to the question in link. The other 2 questions and use-case described have different angles to it. – Vishal Shukla Jan 31 '21 at 16:13

1 Answers1

1

Figured out that XXX-REDUCE-STATE-STORE-changelog topics have cleanup.policy=compact. Meaning the messages will never be deleted as log compaction is enabled. XXX-AGGREGATE-STATE-STORE-repartition topics have retention.ms=-1 by default even if broker level default is set to any other value.

Vishal Shukla
  • 2,848
  • 2
  • 17
  • 19