0

For an ETL job which needs to compute and load over MySQL table rows (At max 30,000 rows with at max 15 small sized columns), which is a better candidate (AWS batch vs Lambda)?

Also want to understand if there any other better options to consider. Would be helpful if one could provide the parameters over which this decision should be made

John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
Aishwer Sharma
  • 205
  • 2
  • 17
  • AWS Lambda have a limit on time period of 15 minutes. I think 15 minutes are more then enough for ETL on 30K records. For now its better to use lambda. As it will take hardly one week for every thing up and running. AWS batch are good when you have long running batch jobs. But development effort is more. There are other options a well like AWS Fargate – Innovation May 11 '21 at 04:14
  • We had a similar discussion, here: https://stackoverflow.com/a/71071271 – ormu5 Mar 27 '23 at 15:03

1 Answers1

0

Batch jobs are used particularly in background work loads while Lambda is useful for near/real time work loads.

Also do consider the business logic complexity while making decision

For more info:

Why/When use batch job: Batch FAQs

Why/When use lambda: Lambda FAQs

Aishwer Sharma
  • 205
  • 2
  • 17