0

I am trying to download multiple files as a zip in Django. It works well when file sizes are small. If the file size is more than 1GB, it's showing memory error:

Traceback (most recent call last):
 x-xxx-xx: File "/usr/lib64/python3.7/zipfile.py", line 1764, in write
 x-xxx-xx: shutil.copyfileobj(src, dest, 1024*8)
 x-xxx-xx: File "/usr/lib64/python3.7/shutil.py", line 82, in copyfileobj
 x-xxx-xx: fdst.write(buf)
 x-xxx-xx: File "/usr/lib64/python3.7/zipfile.py", line 1131, in write
 x-xxx-xx: self._fileobj.write(data)
 x-xxx-xx: MemoryError
 x-xxx-xx: During handling of the above exception, another exception occurred:
 x-xxx-xx: Traceback (most recent call last):
 x-xxx-xx: File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner
 x-xxx-xx: response = get_response(request)
 x-xxx-xx: File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/django/core/handlers/base.py", line 115, in _get_response
 x-xxx-xx: response = self.process_exception_by_middleware(e, request)
 x-xxx-xx: File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/django/core/handlers/base.py", line 113, in _get_response
 x-xxx-xx: response = wrapped_callback(request, *callback_args, **callback_kwargs)
 x-xxx-xx: File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/django/utils/decorators.py", line 130, in _wrapped_view
 x-xxx-xx: response = view_func(request, *args, **kwargs)
 x-xxx-xx: File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/django/views/decorators/cache.py", line 44, in _wrapped_view_func
 x-xxx-xx: response = view_func(request, *args, **kwargs)
 x-xxx-xx: File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/django/contrib/admin/sites.py", line 231, in inner
 x-xxx-xx: return view(request, *args, **kwargs)
 x-xxx-xx: File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/django/views/decorators/cache.py", line 31, in _cache_controlled
 x-xxx-xx: response = viewfunc(request, *args, **kw)
 x-xxx-xx: File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/django/contrib/auth/decorators.py", line 21, in _wrapped_view
 x-xxx-xx: return view_func(request, *args, **kwargs)
 x-xxx-xx: File "/var/app/current/fileupload/views.py", line 519, in downloadScanView
 x-xxx-xx: zf.write(tmp.name, zip_path)
 x-xxx-xx: File "/usr/lib64/python3.7/zipfile.py", line 1764, in write
 x-xxx-xx: shutil.copyfileobj(src, dest, 1024*8)
 x-xxx-xx: File "/usr/lib64/python3.7/zipfile.py", line 1143, in close
 x-xxx-xx: self._fileobj.write(buf)
 x-xxx-xx: ValueError: I/O operation on closed file.

This Django application is up and running in AWS. I understand that the problem arises because of insufficient memory allocation. I am using the following code to serve the files. Is it possible to pass the file content as a chunk to utilize the memory more efficiently? Any help would be appreciated.

zip_subdir = url.split('/')[-1]
zip_filename = zip_subdir + ".zip"
byte_stream = BytesIO()
zf = ZipFile(byte_stream, "w", zipfile.ZIP_DEFLATED)

for path in s3_file_path:
    url = s3Client.generate_presigned_url('get_object', Params = {'Bucket': BUCKET_NAME, 'Key': path.key}, ExpiresIn = 100)
    file_response = requests.get(url)
    if file_response.status_code == 200:
        try:
           tmp = tempfile.NamedTemporaryFile()
           tmp.name = path.key.split('/')[-1]
           f1 = open(tmp.name, 'wb')
           f1.write(file_response.content)
           f1.close()
           zip_path = os.path.join('/'.join(path.key.split('/')[1:-1]), tmp.name)
           zf.write(tmp.name, zip_path)
        finally:
           os.remove(tmp.name)
zf.close()
response = HttpResponse(byte_stream.getvalue(), content_type="application/x-zip-compressed")
response['Content-Disposition'] = 'attachment; filename=%s' % zip_filename
return response
reasm001
  • 167
  • 1
  • 11

0 Answers0