Run AWS lambda function on existing S3 images

Question

I write an AWS lambda function in Node.js for image resizing and trigger it when images upload. I have already more than 1,000,000 images existing in bucket.

I want to run this lambda function on that images but not find anything till yet.

How can I run AWS lamdba function on existing images of S3 bucket?

Note:- I know this question already asked on Stack overflow, but issue is that no solution of them given till yet

score 4 · Answer 1 · answered Jul 04 '18 at 15:14

Unfortunately, Lambda cannot be triggered automatically for objects that are already existing in a S3 bucket.

You will have to invoke your Lambda function manually for each image in your S3 bucket.

First, you will need to list existing objects in your S3 bucket using the ListObjectsV2 action.

For each object in your S3 bucket, you must then invoke your Lambda function and provide the S3 object's information as the Payload.

score 4 · Answer 2 · edited Jun 20 '20 at 09:12

Yes , it's completely true that lambda cannot be triggered by objects already present there in your s3 bucket, but invoking your lambda manually for each object is a completely dumb idea.

With some clever techniques you can perform your tasks on those images easily :

The hard way is, make a program locally that exactly does the same thing as your lambda function but add two more things, firstly you have to iterate over each object in your bucket, then perform your code on it and then save it to destination path of s3 after resizing. i.e, for all images already stored in your s3 bucket , instead of using lambda, you are resizing the images locally in your computer and saving them back to s3 destination.
The easiest way is, first make sure that you have configured s3 notification's event type to be Object Created (All) as trigger for your lambda.

Then after this, move all your already stored images to a new temporary bucket, and then move those images back to the original bucket, this is how your lambda will get triggered for each image automatically. You can do the moving task easily by using sdk's provided by AWS. For example, for moving files using boto3 in python, you can refer this link to moving example in python using boto3

Instead of using moving , i.e cut and paste , you can use copy and paste commands too.

You can , but the second way is easier by moving your images to a temporary bucket and then back to original bucket — Mausam Sharma, Jul 05 '18 at 09:10
Probably not the best method with 1 million existing files, unless they are done in smaller batches. — John Rotenstein, Jul 05 '18 at 11:47

score 2 · Answer 3 · answered Jul 30 '19 at 02:00

In addition to Mausam Sharma's comment you can run the copy between buckets using the aws cli:

aws s3 sync s3://SOURCE-BUCKET-NAME s3://DESTINATION-BUCKET-NAME --source-region SOURCE-REGION-NAME --region DESTINATION-REGION-NAME

from here: https://medium.com/tensult/copy-s3-bucket-objects-across-aws-accounts-e46c15c4b9e1

score 2 · Answer 4 · answered Apr 13 '20 at 14:30

You can simply copy back to the same bucket with the CLI which will replace the original file with itself and then run the lambda as a result.

aws s3 copy s3://SOURCE-BUCKET-NAME s3://SOURCE-BUCKET-NAME --recursive

You can also include/exclude glob patterns which can be used to selectively run against say a particular day, or specific extensions etc.

aws s3 copy s3://SOURCE-BUCKET-NAME s3://SOURCE-BUCKET-NAME --recursive --exclude "*" --include "2020-01-15*"

It's worth noting that like many of the other answers here, this will incur costs on s3 for read/write etc, so cautiously apply this in the event of buckets containing lots of files.

Run AWS lambda function on existing S3 images

4 Answers4