NGinx & Django, serving large files (3gb+)

Question

This question was already answered here.

I'm having some problems to serve large file downloads/uploads (3gb+).

As I'm using Django I guess that the problems to server the file can become from Django or NGinx.

In my NGinx enabled site I have

server {
    ...
    client_max_body_size 4G;
    ...
}

And over django I'm serving the files in chunk sizes:

def return_file(path):
        filename = os.path.basename(path)
        chunk_size = 8192
        response = StreamingHttpResponse(FileWrapper(open(path), chunk_size), content_type=mimetypes.guess_type(path)[0])
        response['Content-Length'] = os.path.getsize(path)    
        response['Content-Disposition'] = 'attachment; filename={0}'.format(filename)
        return response

This method allowed me to pass from downloads of 600Mb~ to 2.6Gb, but it seems that the downloads are getting truncated at 2.6Gb. I traced the error:

2015/09/04 11:31:30 [error] 831#0: *553 upstream prematurely closed connection while reading upstream, client: 127.0.0.1, server: localhost, request: "GET /chat/download/photorec.zip/ HTTP/1.1", upstream: "http://unix:/web/rsmweb/run/gunicorn.sock:/chat/download/photorec.zip/", host: "localhost", referrer: "http://localhost/chat/2/"

After reading some posts I added the following to my NGinx conf:

   proxy_read_timeout 300;
   proxy_connect_timeout 300;
   proxy_redirect off;

But I got the same error with an *1 instead of a *553*

I also thought that It could be a Django database Timeout, so I added:

DATABASE_OPTIONS = {
    'connect_timeout': 14400,
}

But it is not working either. (the download over the development server takes about 30 seconds)

Thanks for any help!

score 2 · Accepted Answer · edited May 23 '17 at 11:44

2

For large files try to use NGINX itself with X-Accel. NGINX is intended to server static content, while Django is for your application logic.

For more information NGINX X-Accel Wiki and this answer.

edited May 23 '17 at 11:44

Community

1
1

answered Sep 22 '15 at 05:37

mergenchik

1,129
16
27

score -1 · Answer 2 · answered Sep 04 '15 at 10:54

-1

The error from nginx indicates that the upstream closed the connection, so it's a problem with django. I'd recommend looking for errors and debugging information in the django logs.

answered Sep 04 '15 at 10:54

womble

12,033
5
52
66

I'll try to find what's the problem with django but it is not loggin/displaying any error! – Sep 04 '15 at 10:59
You'll need to do some debugging, then. I'm not a Python programmer, and also this isn't a programming Q&A site, so there won't be much more help available from me. – womble Sep 04 '15 at 11:06
I thought that this was a question for ServerFault instead of S.O because it is pretty much about the "server settings", but thanks for pointing it, and thanks for your help! – Sep 04 '15 at 11:12

NGinx & Django, serving large files (3gb+)

2 Answers2