0

I want my users to be able to download many files from AWS S3 bucket(potentially over few hundred GBs sized when accumulated) as one large ZIP file. I would download those selected files from S3 first and upload a newly created ZIP file on S3. This job will be rarely invoked during our service, so I decided to use Lambda for it.

But Lambda has its own limitations - 15 min of execution time, ~500MB /tmp storage, etc. I found several workaround solutions on Google that can beat the storage limit(streaming) but found no way to solve execution time limit.

Here are what I've found so far:

Note that programming language is not a concern here.

Could you please give me a suggestion?

hallazzang
  • 651
  • 8
  • 18
  • 1
    Don't use lambda? Seams rather obvious that lambda is not the good match for your requirement. – Marcin Mar 17 '20 at 05:14
  • @Marcin running a separate EC2 instance would fit better for this kind of job? I'm considering about the cost, too. – hallazzang Mar 17 '20 at 05:18
  • ec2 instance or docker on fargate. These are possibilities worth checking out. Have to investigate their advantages/disadvantages for your scenario and decide if its worth pursing lambda solution. Lambda is not a golden solution for every possible use case. – Marcin Mar 17 '20 at 05:24
  • 1
    @Marcin thanks, I'll check those options! – hallazzang Mar 17 '20 at 05:30

0 Answers0