How can I measure the propagation latency of DynamoDB Streams?

Question

I'm using DynamoDB Streams + Kinesis Client Library (KCL). How can I measure latency between when an event was created in a stream and when it was processed on KCL side?

As I know, KCL's MillisBehindLatest metric is specific to Kinesis Streams(not DynamoDB streams). approximateCreationDateTime record attribute has a minute-level approximation, which is not acceptable for monitoring in sub-second latency systems.

Could you please help with some useful metrics for monitoringDynamoDB Streams latency?

Answer 1

You can change the way you do writes in your application to allow your application to track the propagation delay of mutations in the table's stream. For example, you could always update a 'last_updated=' timestamp attribute when you create and update items. That way, when your creations and updates appear in the stream, you can estimate the propagation delay by subtracting the current time from last_updated in the NEW_IMAGE of the stream record.

Because deletions do not have a NEW_IMAGE in stream records, your deletes would need to take place in two steps:

1. logical deletion where you write the 'logically_deleted=' timestamp to the item and
2. physical deletion where you actually call DeleteItem immediately following 1.

Then, you would use the same math as for creations and updates, only differences being that you would use the OLD_IMAGE when processing deletions and you would need to subtract at least around 10ms to account for the time it takes to perform the logical delete (step 1).