1

I'm building a serverless web application using: API-gateway, S3, DynamoDB, Cloud Formation, ... I have come to a point where I'm adding a new feature that requires some data in S3 and DynamoDB to be migrated. I could of course write a one-off script that would do the job, but I was wondering if there are any good structural solutions out there?

I found https://github.com/sequelize/umzug but didn't look any deeper yet. Any reviews?

I'm particularly interested to learn how a data migration might work where there is an infrastructure change, e.g. data in one S3 bucket is migrated into a new S3 Bucket and a DynamoDB table. Anybody has some experience with this?

Many thanks!

Karel
  • 185
  • 2
  • 7

1 Answers1

0

In general we have "Data Migration Service" and "Schema conversion tool" to migrate between different DB engines. Also look at "AWS Data Pipeline" ..

To answer your question; it will depend on your requirements. To give you few examples; you can directly migrate data from one S3 bucket to other using a AWS CLI command "aws s3 cp" if there is no conversion needed in between. for example: You can copy data from S3 to redshift using "copy" command.

Deepak Singhal
  • 10,568
  • 11
  • 59
  • 98
  • 1
    I'm more looking for a solution that can manage the different migrations that are created in the lifetime of an application. I worked with Ruby on Rails in the past which has this type of migration management. – Karel Jul 02 '19 at 12:45
  • are u talking about application migration ( from one technology stack to other ) or data migration (one db store to other ) ? – Deepak Singhal Jul 02 '19 at 15:11
  • 1
    Data migration. e.g. you have a DynamoDB table containing users. Up to now all your users were of the same type, but now you want to have a property type for each user. You could catch it in the code, but it's nice to add property type=standard to the existing users. So you write a small script to do the migration of the DynamoDB table. After some time you have many versions of you schema and many scripts to convert them. RoR has a way to manage these versions. This might be interesting if you want to restore an old backup, ... – Karel Jul 02 '19 at 17:57
  • Got it.. You looking for something like this? https://dbmstools.com/version-control-tools – Deepak Singhal Jul 02 '19 at 18:03
  • Yes these Migration-based is what I mean, but for when the data is in S3 and DynamoDB,... – Karel Jul 03 '19 at 08:32
  • @Karel did you find a solution for this? – fischermatte May 12 '20 at 15:29
  • @fischermatte I'm writing one-off scripts at this point, so not really... – Karel May 12 '20 at 18:31