AWS DynamoDB to S3

Question

I want to move(export) data from DynamoDB to S3

I have seen this tutorial but i'm not sure if the extracted data of dynamoDB will be deleted or coexits in DynamoDB and S3 at the same time.

What I expect is the data from dynamoDB will be deleted and stored in s3 (after X time stored in DynamoDB)

The main purpose of the project could be similar to this

There are any way to do this without have to develop a lambda function?

In resume, I have found this 2 different ways:

• DynamoDB -> Pipeline -> S3 (Are the dynamoDB data deleted?)

• DynamoDB -> TTL DynamoDB + DynamoDB stream -> Lambda -> firehose -> s3 (this appears to be more difficult)

Is this post currently valid for this purpouse?

What would be the simpliest and fasted way?

Answer 1

In your first option, as per default, data is not removed from dynamoDB. You can design a pipeline to make this work, but I think that is not the best solution.

In your second option, you must evaluate the solution based on your expected data volume:

1. If the data volume that will expire in TTL definition is not very large, you can use lambda to persist removed data into S3 without Firehose. You can design a simple lambda function to be triggered by DynamoDB Stream and persist each stream event as a S3 object. You can even trigger another lambda function to consolidate the objects in a single file in the end of the day, week or month. But again, based on your expected volume.

2. If you have a lot of data being expired at the same time and you must perform transformations on this data, the best solution is to use Firehose. Firehose can proceed with the transformation, encryption and compact your data before sending it to S3. If the volume of data is to big, using functions in the end of the day, week or month may not be feasible. So it's better to perform all this procedures before persisting it.

Answer 2

You can use AWS Pipeline to dump DynamoDB table to S3 and it will not be deleted.