Django 1.11 download file chunk by chunk

Question

In my case, I have the Django 1.11 server acting as a proxy. When you click "download" from the browser, it sends a request to the django proxy that downloads files from another server and processes them, after which they must "send" them to the browser to allow the user to download them. My proxy downloads and processes the files chunks by chunks. How can I send chunks to the browser as they are ready so that the user finally downloads a single file?

In practice, I have to let you download a file that is not yet ready, like a stream.

def my_download(self, res)

   # some code
   file_handle = open(local_path, 'wb', self.chunk_size)

   for chunk in res.iter_content(self.chunk_size):
        i = i+1
        print("index: ", i, "/", chunks)
        if i > chunks-1:
            is_last = True

        # some code on the chunk

        # Here, instead of saving the chunk locally, I would like to allow it to download it directly.
        file_handle.write(chunk)
    file_handle.close()

    return True

Thank you in advance, greetings.

I finally found the answer here: https://stackoverflow.com/questions/38514919/django-stream-request-from-external-site-as-received This question is actually a dupplicate — Luis Sieira, Mar 02 '18 at 13:30
yeah, and here https://stackoverflow.com/questions/48949022/django-filewrapper-memory-error-serving-big-files-how-to-stream/48949959#48949959 — trinchet, Mar 02 '18 at 13:45
And here: https://stackoverflow.com/questions/8600843/serving-large-files-with-high-loads-in-django?answertab=votes#tab-top — John Moutafis, Mar 02 '18 at 13:59
Possible duplicate of [Django - Stream request from external site as received](https://stackoverflow.com/questions/38514919/django-stream-request-from-external-site-as-received) — Luis Sieira, Mar 05 '18 at 13:55
Possible duplicate of [Serving large files ( with high loads ) in Django](https://stackoverflow.com/questions/8600843/serving-large-files-with-high-loads-in-django) — John Moutafis, Mar 05 '18 at 18:08

John Moutafis · Answer 1 · 2018-03-02T14:16:53.387

This question should be flagged as duplicate of this post: Serving large files ( with high loads ) in Django

Always try to find the answer before you create a question in SO, please!

Essentially the answer is included in Django's Documentation: "Streaming Large CSV files" example and we will apply the above question into that example:

You can use Django's StreamingHttpResponse and Python's wsgiref.util.FileWrapper to serve a large file in chunks effectivelly and without loading it in memory.

def my_download(request):
    file_path = 'path/to/file'
    chunk_size = DEFINE_A_CHUNK_SIZE_AS_INTEGER
    filename = os.path.basename(file_path)

    response = StreamingHttpResponse(
        FileWrapper(open(file_path, 'rb'), chunk_size),
        content_type="application/octet-stream"
    )
    response['Content-Length'] = os.path.getsize(file_path)    
    response['Content-Disposition'] = "attachment; filename=%s" % filename
    return response

Now if you want to apply some processing to the file chunk-by-chunk you can utilize FileWrapper's generated iterator:

Place your chunk processing code in a function which MUST return the chunk:

def chunk_processing(chunk):
    # Process your chunk here
    # Be careful to preserve chunk's initial size. 
    return processed_chunk

Now apply the function inside the StreamingHttpResponse:

response = StreamingHttpResponse(
    (
        process_chunk(chunk) 
        for chunk in FileWrapper(open(file_path, 'rb'), chunk_size
    ),content_type="application/octet-stream"
)

I was looking for that answer some minutes before finding this question (that's why I proposed a bounty). Actually "serving large files" is not the only use I can think of to this, so I didn't consider looking for it like that — Luis Sieira, Mar 02 '18 at 14:02
@LuisSieira I edited my answer a bit to include a part of the answer to OP's initial question. Nevertheless, this is not far from a duplicate question... — John Moutafis, Mar 02 '18 at 14:18
Yes, and it should be flagged like so, but it's useful as a reformulation of the problem that addresses to the previous questions answers. — Luis Sieira, Mar 02 '18 at 14:19

Django 1.11 download file chunk by chunk

1 Answers1