Synchronizing S3 Folders/Buckets

Question

I have an S3 Bucket that holds static content for all my clients in production. I also have a staging environment which I use for testing before I deploy to production. I also want the staging environment to point to S3 to test uploads and other functions. Problem is, I don't want the staging server to reference the same production s3 bucket/folder, because there is a risk of overriding production files.

My solution is to use a different folder within the same bucket, or create a different bucket all together that I can refresh periodically with the contents of the production bucket. Is there a way to easily sync two folders or buckets on Amazon S3?

Any other suggestions for managing this type of scenario would also be greatly appreciated.

Having tried all solutions for sync to S3 I find https://www.syncovery.com/ the most full-featured and reliable. Hope it helps someone. — Vacilando, Oct 29 '14 at 08:25
see also https://stackoverflow.com/questions/26326408/difference-between-s3cmd-boto-and-aws-cli e.g., `aws s3 sync my-dir s3://my-bucket` or `aws s3 sync s3://my-bucket/my-folder2/ my-dir` etc etc — michael, Nov 04 '17 at 10:24

score 9 · Answer 1 · answered Jul 06 '10 at 13:04

9

s3cmd is a nice CLI utility you can use in a cronjob. It even as a sync feature similar to *nix rsync.

answered Jul 06 '10 at 13:04

Jay Sidri

6,271
3
43
62

score 4 · Answer 2 · answered Jan 27 '12 at 21:18

4

There's also DragonDisk - like cloudberry explorer and other amazon s3 clients - except it's free and multi-platform (QT).

It does sync jobs, both local<->s3 and s3<->s3. It also has a command line interface which can do syncing: http://www.dragondisk.com/faq.html

answered Jan 27 '12 at 21:18

Duncan Lock

12,351
5
40
47

On OSX, wow, is it ugly. But it does sync perfectly. – Justin Feb 28 '14 at 16:32

admon_org · Answer 3 · 2017-12-06T06:09:51.757

2

Here's an example on S3CMD for your reference https://www.admon.org/system-tuning/sync-two-amazon-s3-buckets/

edited Dec 06 '17 at 06:09

answered Aug 03 '11 at 02:27

admon_org

39
3

1

Dead link. This is why we don't just paste links into SO answers. – michael Nov 04 '17 at 10:20

score 1 · Answer 4 · answered Dec 06 '09 at 11:45

1

Check out CloudBerry Explorer and it's ability to sync data between local computer and Amazon S3. Might not exactly what you want but will help you to get started. More info here

answered Dec 06 '09 at 11:45

cloudberryman

4,598
2
27
14

Funny thing you mentioned this. It's exactly what I ended up using :) It still would be nice to have a command line tool that I could execute with a cron job since I have more than 20,000 files! – Zakir Hemraj Dec 06 '09 at 21:34
you can use CloudBerry Explorer PowerShell interface to automate the whole stuff – cloudberryman Dec 19 '09 at 09:39
See answer below (s3cmd) if you're looking to automate syncing via a shell script etc – Jay Sidri Aug 09 '12 at 05:05

score 0 · Accepted Answer · answered Dec 14 '09 at 08:58

0

CloudBerry Explorer comes with PowerShell command line interface and you can learn here how to use it to do sync.

answered Dec 14 '09 at 08:58

cloudberryman

4,598
2
27
14

2

Looks like CloudBerry is only for windows, do you have any mac recommendations? – Schneems Apr 05 '12 at 03:43
Just note that CloudBerry doesn't actually sync file headers (this functionality just doesn't work) – Alex LaWay Jan 09 '14 at 12:06

Synchronizing S3 Folders/Buckets

5 Answers5