Pull-style cross region replication for S3 buckets

Question

I need to pull data published to an S3 bucket by a different organization (therefore a different AWS account) in a different region, for subsequent processing with Lambda. I do have access to read it but cannot ask them to set up replication to my buckets.

Amazon's Cross-Region Replication looks like it's designed for pushing data from the source and I'm not even sure the source organization has versioning enabled.

Is there a way to pull data? My need is for one-way only; I need to process that data shortly (within 10 minutes or so) after it arrives in the source S3 bucket.

A cron-job that runs `aws s3 sync` every 10 minutes? Something like that is going to be the best way to pull from an S3 bucket I think, if you can't get new object events sent to you from that bucket. — Mark B, Jan 14 '19 at 15:35
Is there a way to run this as a lambda? I'm thinking of the cost of running an EC2 instance just to run the sync. Thanks. — wishihadabettername, Jan 14 '19 at 15:40

score 2 · Accepted Answer · answered Jan 14 '19 at 15:50

2

You could run aws s3 sync on a schedule, like every 10 minutes. If you want to run this in a AWS Lambda function, it looks like NodeJS and Python Lambda environments have the AWS CLI tool pre-installed. I would suggest writing a short Python Lambda function that calls the AWS CLI took to run an s3 sync command, and schedule that Lambda function to run every 10 minutes.

answered Jan 14 '19 at 15:50

Mark B

183,023
24
297
295

1

I would use a CloudWatch Rule to trigger the lambda on a schedule – David Webster Jan 14 '19 at 15:54
@DavidWebster yes that is how you schedule a Lambda function – Mark B Jan 14 '19 at 15:59
1

Thanks. I'll give it a try the moment I can and then will accept the answer (with any details that might be useful to others who might visit the page). – wishihadabettername Jan 14 '19 at 18:27

Pull-style cross region replication for S3 buckets

1 Answers1

Linked