20

I'm considering using Apache Kafka as an event store for storing events within a microservice.

One thing that I read through various blogs is that Kafka can be considered to be a single source of truth, where Kafka log will store all the events for a given topic.

I was wondering if Kafka has the ability to replay messages since the beginning of time (in case there is a hard drive/network crash that occurs for example)?

(note that i see that there are some logs stored in the /tmp folder under a topic directory). Does anyone know of any command (if any) that can be invoked to replay the messages in the topic?

stefanobaghino
  • 11,253
  • 4
  • 35
  • 63
rm12345
  • 1,089
  • 3
  • 18
  • 32

2 Answers2

11

Yes, you can seek to a specific offset, but

beginning of time

depends on the topic or broker configuration. IIRC, the default retention is 7 days.

Refer to the the Kafka documentation.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Gary Russell
  • 166,535
  • 14
  • 146
  • 179
  • 1
    Thank you for the useful answer! The link points to all of the documentation at the moment. It might be more helpful (especially for beginners) if it pointed to a specific section related to your answer. Thanks again! – nkhil Jul 29 '20 at 08:29
  • 1
    Back then, there were no anchors on specific properties; now there is - https://kafka.apache.org/documentation/#log.retention.hours and https://kafka.apache.org/documentation/#retention.ms - also https://kafka.apache.org/documentation/#log.retention.bytes and https://kafka.apache.org/documentation/#retention.bytes – Gary Russell Jul 29 '20 at 13:12
4

Yes, You can replay message. As Consumer have a control over resetting the offset. You can start reading messages from the beginning or if you know any existing offset value you can read it from there as well. Once the message is committed it will be in there in topic until its retention period is over. Default retention period is 7 days, however you can change it any time.

Vishal Akkalkote
  • 1,151
  • 12
  • 16