0

I would like to understand what are the best ways of handling a kinesis consumer outage.

My consumer is set up with TRIM_HORIZON. Below is the definition from AWS:

TRIM_HORIZON - Start reading at the last (untrimmed) stream record, which is the oldest record in the shard. In DynamoDB Streams, there is a 24 hour limit on data retention. Stream records whose age exceeds this limit are subject to removal (trimming) from the stream.

My understanding from this definition is that if my consumer is down and then restarted. It will start consuming from the oldest readable record in the shard. It means that it will re-read some old data.

I would like that my consumer starts after the last acknowledge record.

My kinesis consumer uses KCL (java) and is backed by dynamoDB.

Should I use AFTER_SEQUENCE_NUMBER and give my consumer a sequence number after an outage? How do I get the last read recored? My goal is to resume consuming records from the last acknowledge record before the consumer downtime?

I appreciate your recommendations.

Thanks,

Moulay.

moulay
  • 1
  • Also consider referring to [When AWS KCL processRecords is failed, how to “mark” that the records should be reprocessed?](https://stackoverflow.com/questions/43046982/when-aws-kcl-processrecords-is-failed-how-to-mark-that-the-records-should-be) – Krease Apr 23 '18 at 23:56
  • Thanks @Krease for referring to the other questions. I am closing this one. – moulay Apr 26 '18 at 18:32

0 Answers0