0

Hope everybody is safe and happy.

I have given a AWS task as follow.

Requirement: As soon as file gets uploaded in s3 bucket,it should be zipped and must be uploaded back to another s3 bucket.

I am able to complete this task using lambda(python) but it involves lots of disk IO. So ,i am looking for a solution where i will not require to store incoming s3 object at /tmp/ folder. As we all know , lambda provides very less memory and storage(500MB). Hence, wants to avoid this approach.

So, does anybody aware of how to zip the incoming s3 file on the fly? I am only aware that, it possible to do with the help of streaming of s3 object. But not able to find how it can be achieved end to end using python language.

Thanks in advance.

Manoj Dhake
  • 227
  • 4
  • 16
  • 1
    This is called streaming. Search for Python zip streaming solutions and learn about the boto3 client's streaming support on `put_object()`. – jarmod Oct 30 '20 at 12:57

1 Answers1

0

You can use python generators in the Lambda to zip the s3 files and put the zip file back in another s3 as the generators doesn't occupy memory.

Dharman
  • 30,962
  • 25
  • 85
  • 135
Pendyala Narendra
  • 1
  • 1
  • Do you have any reference link for python generators? And does python generator do not require to store files at /tmp/ location? Is this solution best for large s3 file as well? – Manoj Dhake Oct 30 '20 at 12:26
  • Please go through this link:https://stackoverflow.com/questions/297345/create-a-zip-file-from-a-generator-in-python – Pendyala Narendra Oct 30 '20 at 12:34
  • Thanks Narendra. Actually, i am begineer to aws and python. Is there any step by step document or link where everything is explained properly about this solution along with aws lambda? – Manoj Dhake Oct 30 '20 at 13:51
  • I will help you surely – Pendyala Narendra Oct 30 '20 at 16:26
  • Hi Narendra. Thanks again for giving reply. Would you please guide me how can i start understanding the code which is given in that link? What would be my next step to complete the task? – Manoj Dhake Oct 31 '20 at 17:05