1

I want to run a parallel task with a step function on a really big file.

is there a way to make the lambdas that will be processing the big file receive a different part of the file? Instead of them all getting the same big file?

WeCanBeFriends
  • 641
  • 1
  • 10
  • 23
  • all plain text files? For example, you can split it and handle 100 lines only each time. – BMW Feb 16 '18 at 03:55
  • It's a csv file, how do I pass the seperate inputs to each lambda to process in parallel? – WeCanBeFriends Feb 16 '18 at 12:25
  • 3
    You will need either to split the file and create multiple smaller files on S3 or do partial reads from S3 in each Lambda: https://stackoverflow.com/questions/36436057/s3-how-to-do-a-partial-read-seek-without-downloading-the-complete-file – FelixEnescu Feb 16 '18 at 16:08
  • Thanks blueCat. If you write that up, I will accept as the answer. – WeCanBeFriends Feb 16 '18 at 23:26

0 Answers0