3

We have multiple buckets that are used by our clients. A client uploads a single file (random filename) to their bucket, and we then visit that bucket and copy it to our own bucket for processing. Basically, this: https://stackoverflow.com/a/10418427/2868238

How could I automate this? I note lambda has s3 object event support so wonder if I can use this somehow?

Paul.

Community
  • 1
  • 1
Paul
  • 578
  • 1
  • 8
  • 23

2 Answers2

7

This use case is almost the textbook example for AWS Lambda. If you look at the AWS Lambda image resize example, all you need to do is remove code that tests for the image type and actually does the resize- it's designed to download, transform, then upload the object to a new S3 bucket.

Also, you may be able to do this even more easily (and cheaply) with S3 cross-region replication, but that requires the buckets be in different regions (thanks @William-Gaul).

So, it depends on your precise use case.

tedder42
  • 23,519
  • 13
  • 86
  • 102
  • 2
    Beat me to it! One caveat for the new S3 replication feature though: "Replication always takes place between a pair of AWS regions. You cannot use this feature to replicate content to two buckets that are in the same region." – William Gaul Mar 25 '15 at 16:10
  • 3
    I'd just add that S3 replication can only copy to one other bucket, not to multiple buckets. (OP doesn't need multiple buckets, but someone reading this might.) – ron rothman Aug 09 '16 at 18:40
  • I'll throw a warning out there. I noticed that our cross region replication was taking 3 minutes last week with very small files. Previously I had seen replication in seconds. I reached out to AWS Support to see if there was something going on or, if not, what we should expect. We were told that there is no SLA for the CRR feature and that replication could take up to hours. The workaround they offered - S3 events triggering Lambda copy functions. – Tom Sep 28 '16 at 02:35
  • Anyone have any idea how to accomplish an `s3.copyObject` to a bucket under another account using a Lambda function? See this question for more details: http://stackoverflow.com/questions/40187728/how-to-use-aws-lambda-to-backup-an-s3-object-to-a-bucket-on-another-account – AJB Oct 22 '16 at 02:50
  • Another warning RE cross-region replication; it will only work if both source & destination buckets have versioning enabled. Which can throw a spanner in the works - like it did for us. – Steve Horvath Jul 20 '17 at 04:11
  • aws now supports same bucket region policy replication. See here https://aws.amazon.com/about-aws/whats-new/2019/09/amazon-s3-introduces-same-region-replication/ – george Sep 30 '19 at 14:40
1

This should help you out , it is written in detail on how to do replication between two buckets based on events. http://docs.aws.amazon.com/lambda/latest/dg/with-s3-example-prepare.html

Joel Nishanth Ponukumatla
  • 642
  • 6
  • 8