Stack Overflow
Stack Overflow

Questions tagged [amazon-dynamodb-streams]

Use this tag for questions related to DynamoDB Streams, which provides a time ordered sequence of item level changes in any DynamoDB table. The changes are de-duplicated and stored for 24 hours. This capability enables you to extend the power of DynamoDB with cross-region replication, continuous analytics with Redshift integration, change notifications, and many other scenarios.

Useful links:

  1. Documentation
  2. Use cases
393 questions
49
votes
3 answers

How to do basic aggregation with DynamoDB?

How is aggregation achieved with dynamodb? Mongodb and couchbase have map reduce support. Lets say we are building a tech blog where users can post articles. And say articles can be tagged. user { id : 1235, name : "John", …
prem kumar
  • 5,641
  • 3
  • 24
  • 36
42
votes
3 answers

Difference between Kinesis Stream and DynamoDB streams

They seem to be doing the same thing to me. Can anyone explain to me the difference?
Junji Zhi
  • 1,382
  • 1
  • 14
  • 22
24
votes
5 answers

AWS Glue: How to handle nested JSON with varying schemas

Objective: We're hoping to use the AWS Glue Data Catalog to create a single table for JSON data residing in an S3 bucket, which we would then query and parse via Redshift Spectrum. Background: The JSON data is from DynamoDB Streams and is deeply…
ehelander
  • 243
  • 1
  • 2
  • 5
23
votes
1 answer

Dynamodb streams in python

I would like to read data from a dynamodb stream in python and the alternatives that i have found so far are Use dynamodb stream low level library functions (as described here): This solution however seems almost impossible to maintain in a…
Ashish
  • 231
  • 2
  • 3
20
votes
1 answer

Stream support for local dynamodb?

I can't seem to get stream support working in dynamo db local, are they supported? The only indication I could find that they are, is the very last bullet point in the developer guide regarding local differences: If you're using DynamoDB Streams,…
jschr
  • 1,866
  • 2
  • 16
  • 21
19
votes
2 answers

Multiple AWS Lambda functions on a Single DynamoDB Stream

I have a Lambda function to which multiple DynamoDB streams are configured as event sources and this is a part of a bigger pipeline. While doing my checks, I found some missing data in one of the downstream components. I want to write a simpler…
witchking
  • 365
  • 1
  • 3
  • 9
18
votes
3 answers

How do DynamoDB streams distribute records to shards?

My goal is to ensure that records published by a DynamoDB stream are processed in the "correct" order. My table contains events for customers. Hash key is Event ID, range key a timestamp. "Correct" order would mean that events for the same customer…
EagleBeak
  • 6,939
  • 8
  • 31
  • 47
17
votes
3 answers

DynamoDb : Scan query does not return all the data

I have a DynamoDb table with thousands of data. I am scanning the table using Scan function and I have applied "Between" FilterExpression. However , the query response only gives 3 records whereas it should return about 100 records. I have created…
Sumera
  • 205
  • 1
  • 2
  • 9
17
votes
8 answers

How to get the pure Json string from DynamoDB stream new image?

I've a Dynamodb table with streaming enabled. Also I've created a trigger for this table which calls an AWS Lambda function. Within this lambda function, I'm trying read the new image (Dynamodb item after the modification) from the Dynamodb stream…
Asanga Dewaguru
  • 1,058
  • 2
  • 16
  • 31
16
votes
3 answers

DynamoDB : List all partition keys

I want to update contains in DynamoDB, for which I need to iterate over existing partition keys present in table. Is there any way to fetch only list of partition keys using Python. Scan and Query only work on attributes of my table. Is there any…
WPrathamesh
  • 175
  • 1
  • 1
  • 5
15
votes
1 answer

Creation of GSI taking long time

I have a table with close to 2 billion rows already created in DynamoDB. Due to a query requirement, I had to create a Global Secondary Index(GSI) in it. The process of GSI creation started 36 hours ago but still isn't completed. Portal shows Item…
Dixit Gokhale
  • 601
  • 1
  • 12
  • 32
13
votes
2 answers

How to reference the latest Stream of a DynamoDB table in a Cloudformation template

I am writing a plugin for the serverless framework, which references a DynamoDB Stream by its ARN. I can construct the DynamoDB tables ARN with the information I have at hand, but I don't know the timestamp part, that would be necessary to build the…
Thomas
  • 10,289
  • 13
  • 39
  • 55
12
votes
3 answers

AWS Lambda processing stream from DynamoDB

I'm trying to create a lambda function that is consuming a stream from dynamoDB table. However I was wondering which is the best practice to handle data that may not have been processed for some errors during the execution? For example my lambda…
Simone Tallevi-Diotallevi
  • 153
  • 1
  • 1
  • 9
12
votes
2 answers

DynamoDB GSI BatchGetItem

Is it possible to retrieve rows from the dynamodb Global secondary index using batchgetitem api? If my aim is to retrieve data from the main table based on some non-key attribute also , but data should be retrieved in the batch of 100 items - is the…
user1846749
  • 2,165
  • 3
  • 23
  • 36
11
votes
1 answer

Incompatible types of AttributeValue in Dynamodb Streams

I am trying to unmarshal a dynamodb stream record with help of the official unmarshall function available in @aws-sdk/util-dynamodb. I am doing this in typescript and the definition looks like this unmarshall: (data: Record,…
Hitesh Balwani
  • 175
  • 1
  • 12
1
2 3
…
26 27