I am trying to download multiple files as a zip in Django. It works well when file sizes are small. If the file size is more than 1GB, it's showing memory error:
Traceback (most recent call last):
x-xxx-xx: File "/usr/lib64/python3.7/zipfile.py", line 1764, in write
x-xxx-xx: shutil.copyfileobj(src, dest, 1024*8)
x-xxx-xx: File "/usr/lib64/python3.7/shutil.py", line 82, in copyfileobj
x-xxx-xx: fdst.write(buf)
x-xxx-xx: File "/usr/lib64/python3.7/zipfile.py", line 1131, in write
x-xxx-xx: self._fileobj.write(data)
x-xxx-xx: MemoryError
x-xxx-xx: During handling of the above exception, another exception occurred:
x-xxx-xx: Traceback (most recent call last):
x-xxx-xx: File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner
x-xxx-xx: response = get_response(request)
x-xxx-xx: File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/django/core/handlers/base.py", line 115, in _get_response
x-xxx-xx: response = self.process_exception_by_middleware(e, request)
x-xxx-xx: File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/django/core/handlers/base.py", line 113, in _get_response
x-xxx-xx: response = wrapped_callback(request, *callback_args, **callback_kwargs)
x-xxx-xx: File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/django/utils/decorators.py", line 130, in _wrapped_view
x-xxx-xx: response = view_func(request, *args, **kwargs)
x-xxx-xx: File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/django/views/decorators/cache.py", line 44, in _wrapped_view_func
x-xxx-xx: response = view_func(request, *args, **kwargs)
x-xxx-xx: File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/django/contrib/admin/sites.py", line 231, in inner
x-xxx-xx: return view(request, *args, **kwargs)
x-xxx-xx: File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/django/views/decorators/cache.py", line 31, in _cache_controlled
x-xxx-xx: response = viewfunc(request, *args, **kw)
x-xxx-xx: File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/django/contrib/auth/decorators.py", line 21, in _wrapped_view
x-xxx-xx: return view_func(request, *args, **kwargs)
x-xxx-xx: File "/var/app/current/fileupload/views.py", line 519, in downloadScanView
x-xxx-xx: zf.write(tmp.name, zip_path)
x-xxx-xx: File "/usr/lib64/python3.7/zipfile.py", line 1764, in write
x-xxx-xx: shutil.copyfileobj(src, dest, 1024*8)
x-xxx-xx: File "/usr/lib64/python3.7/zipfile.py", line 1143, in close
x-xxx-xx: self._fileobj.write(buf)
x-xxx-xx: ValueError: I/O operation on closed file.
This Django application is up and running in AWS. I understand that the problem arises because of insufficient memory allocation. I am using the following code to serve the files. Is it possible to pass the file content as a chunk to utilize the memory more efficiently? Any help would be appreciated.
zip_subdir = url.split('/')[-1]
zip_filename = zip_subdir + ".zip"
byte_stream = BytesIO()
zf = ZipFile(byte_stream, "w", zipfile.ZIP_DEFLATED)
for path in s3_file_path:
url = s3Client.generate_presigned_url('get_object', Params = {'Bucket': BUCKET_NAME, 'Key': path.key}, ExpiresIn = 100)
file_response = requests.get(url)
if file_response.status_code == 200:
try:
tmp = tempfile.NamedTemporaryFile()
tmp.name = path.key.split('/')[-1]
f1 = open(tmp.name, 'wb')
f1.write(file_response.content)
f1.close()
zip_path = os.path.join('/'.join(path.key.split('/')[1:-1]), tmp.name)
zf.write(tmp.name, zip_path)
finally:
os.remove(tmp.name)
zf.close()
response = HttpResponse(byte_stream.getvalue(), content_type="application/x-zip-compressed")
response['Content-Disposition'] = 'attachment; filename=%s' % zip_filename
return response