3

I am trying to use AWS Kinesis stream for one of our data streams. I would like to monitor pending messages on my stream for ops purposes(scale downstream according to backlog), but unable to find any API that gives (approx) pending messages in my stream.

This looks strange as messages get expired after 7 days and if the producers and consumers are isolated and can't communicate, how do you know messages are expiring. How do you handle this problem?

Thanks!

Idonthaveaname jones
  • 33
  • 3

1 Answers1

2

There is no such concept as "pending" message in Kinesis. All the incoming data will be placed on a shard.

Your consumer application should be in running state all the time, to keep track of changes in your stream. The application (with the help of KCL) will continue to poll "Shard Iterator" in the background, thus you will be notified about the new data when it comes.

Roughly; you can see Kinesis as a FIFO queue and the messages will disappear in a short time if you don't pop them.

If your application will process a few messages in an hour, you should think about changing your architecture. Kinesis is probably not the correct tool for you.

az3
  • 3,571
  • 31
  • 31
  • Thanks for detailed explanation. That makes sense. In my usecase, I get a stream of keys on kinesis stream (few k-TPS) that I want apply some transformations and deliver to another service. I am mainly worried of "silently losing messages" in cases, say increased target service latency or upstream spike creating a backlog. Probably I should be provisioned at max capacity per shard then. – Idonthaveaname jones Feb 24 '16 at 09:39
  • In worker.java , it calls runProcessLoop and in that it calls shardConsumer.consumeShard() there it calls checkAndSubmitNextTask() in that it checks readyForNextTask or not . If notReady it does not consumer new records . So how is it possible worker retrieves new records without recordprocessor process previous ones. – user1846749 Jan 24 '17 at 00:05