1

I am looking into dynamo db streaming options. There are 2:

  1. Dynamo streams
  2. Kinesis streams

Dynamo streams can have a lambda trigger that will consume the stream.

Kinesis streams can also have the lambda trigger that will consume the stream.

In both above cases, in the lambda settings we can choose the batch size.

In dynamodb, what is the difference between using dynamo streams + lambda trigger versus kinesis streams + lambda trigger?

variable
  • 8,262
  • 9
  • 95
  • 215
  • 1
    The most important difference is that dynamo streams guarantee **no duplicates**, while kinesis streams do not. – Marcin Nov 18 '22 at 05:56
  • By no duplicates do you mean atmost once delivery? – variable Nov 18 '22 at 06:01
  • 1
    Yes. This for many applications can be very important. – Marcin Nov 18 '22 at 06:02
  • This would be helpful https://stackoverflow.com/questions/38571922/difference-between-kinesis-stream-and-dynamodb-streams – omuthu Nov 18 '22 at 08:02
  • 1
    also one good thing about DynamoDB stream is that you can select what should be streamed. Like, if you update an item, you can chose to only stream the new value, or old or both. Which could be interesting in case you want to avoid processing same messages again. If an update doesn't change the content of the row, then no event will be sent to the stream. It is a way to do deduplication in iot use case. – brushtakopo Nov 18 '22 at 09:41
  • Dynamodb kinesis stream doesn't allow you this option? – variable Nov 18 '22 at 10:37

1 Answers1

3

Below is some of the key differences between Kinesis Data Streams and DynamoDB Streams. In regards to your question around a Lambda consumer, the important details to you is data retention, ordering, de-duplication and number of allowed consumers.

Properties Kinesis Data Streams for DynamoDB DynamoDB Streams
Data retention Up to 1 year. 24 hours.
Kinesis Client Library (KCL) support Supports KCL versions 1.X and 2.X. Supports KCL version 1.X.
Number of consumers Up to 5 simultaneous consumers per shard, or up to 20 simultaneous consumers per shard with enhanced fan-out. Up to 2 simultaneous consumers per shard.
Throughput quotas Unlimited. Subject to throughput quotas by DynamoDB table and AWS Region.
Record delivery model Pull model over HTTP using GetRecords and with enhanced fan-out, Kinesis Data Streams pushes the records over HTTP/2 by using SubscribeToShard. Pull model over HTTP using GetRecords.
Ordering of records The timestamp attribute on each stream record can be used to identify the actual order in which changes occurred in the DynamoDB table. For each item that is modified in a DynamoDB table, the stream records appear in the same sequence as the actual modifications to the item.
Duplicate records Duplicate records might occasionally appear in the stream. No duplicate records appear in the stream.
Stream processing options Process stream records using AWS Lambda, Kinesis Data Analytics, Kinesis data firehose , or AWS Glue streaming ETL. Process stream records using AWS Lambda or DynamoDB Streams Kinesis adapter.
Durability level Availability zones to provide automatic failover without interruption. Availability zones to provide automatic failover without interruption.
Leeroy Hannigan
  • 11,409
  • 3
  • 14
  • 31