2

I am interested in doing automated real-time data processing on AWS using Lambda and I am not certain about how I can trigger my Lambda function. My data processing code involves taking multiple files and concatenating them into a single data frame after performing calculations on each file. Since files are uploaded simultaneously onto S3 and files are dependent on each other, I would like the Lambda to be only triggered when all files are uploaded.

Current Approaches/Attempts:

-I am considering an S3 trigger, but my concern is that an S3 Trigger will result in an error in the case where a single file upload triggers the Lambda to start. An alternate option would be adding a wait time but that is not preferred to limit the computation resources used.

-A scheduled trigger using Cloudwatch/EventBridge, but this would not be real-time processing.

-SNS trigger, but I am not certain if the message can be automated without knowing the completion in file uploads.

Any suggestion is appreciated! Thank you!

lqw1001
  • 21
  • 1

3 Answers3

0

If you really cannot do it with a scheduled function, the best option is to trigger a Lambda function when an object is created.

The tricky bit is that it will fire your function on each object upload. So you either can identify the "last part", e.g., based on some meta data, or you will need to store and track the state of all uploads, e.g. in a DynamoDB, and do the actual processing only when a batch is complete.

Best, Stefan

StefanN
  • 527
  • 1
  • 4
  • 12
0

Your file coming in parts might be named as -

filename_part1.ext
filename_part2.ext

If any of your systems is generating those files, then use the system to generate a final dummy blank file name as -

filename.final

Since in your S3 event trigger you can use a suffix to generate an event, use .final extension to invoke lambda, and process records.

In an alternative approach, if you do not have access to the server putting objects to your s3 bucket, then with each PUT operation in your s3 bucket, invoke the lambda and insert an entry in dynamoDB. You need to put a unique entry per file (not file parts) in dynamo with -

filename and last_part_recieved_time

The last_part_recieved_time keeps getting updated till you keep getting the file parts.

Now, this table can be looked up by a cron lambda invocation which checks if the time skew (time difference between SYSTIME of lambda invocation and dynamoDB entry - last_part_recieved_time) is enough to process the records.

I will still prefer to go with the first approach as the second one still has a chance for error.

Dev Utkarsh
  • 1,377
  • 2
  • 18
  • 43
0

Since you want this to be as real time as possible, perhaps you could just perform your logic every single time a file is uploaded, updating the version of the output as new files are added, and iterating through an S3 prefix per grouping of files, like in this other SO answer.

In terms of the architecture, you could add in an SQS queue or two to make this more resilient. An S3 Put Event can trigger an SQS message, which can trigger a Lambda function, and you can have error handling logic in the Lambda function that puts that event in a secondary queue with a visibility timeout (sort of like a backoff strategy) or back in the same queue for retries.

Yann Stoneman
  • 953
  • 11
  • 35