NGinx & Django, serving large files (3gb+)

Question

I'm having some problems to serve large file downloads/uploads (3gb+).

As I'm using Django I guess that the problem to serve the file can become from Django or NGinx.

In my NGinx enabled site I have

server {
    ...
    client_max_body_size 4G;
    ...
}

And at django I'm serving the files in chunk sizes:

def return_file(path):
        filename = os.path.basename(path)
        chunk_size = 8192
        response = StreamingHttpResponse(FileWrapper(open(path), chunk_size), content_type=mimetypes.guess_type(path)[0])
        response['Content-Length'] = os.path.getsize(path)    
        response['Content-Disposition'] = 'attachment; filename={0}'.format(filename)
        return response

This method allowed me to pass from downloads of 600Mb~ to 2.6Gb, but it seems that the downloads are getting truncated at 2.6Gb. I traced the error:

2015/09/04 11:31:30 [error] 831#0: *553 upstream prematurely closed connection while reading upstream, client: 127.0.0.1, server: localhost, request: "GET /chat/download/photorec.zip/ HTTP/1.1", upstream: "http://unix:/web/rsmweb/run/gunicorn.sock:/chat/download/photorec.zip/", host: "localhost", referrer: "http://localhost/chat/2/"

After reading some posts I added the following to my NGinx conf:

   proxy_read_timeout 300;
   proxy_connect_timeout 300;
   proxy_redirect off;

But I got the same error with an *1 instead of a *553*

I also thought that It could be a Django database Timeout, so I added:

DATABASE_OPTIONS = {
    'connect_timeout': 14400,
}

But it is not working either. (the download over the development server takes about 30 seconds)

PS: Some one already pointed me that the problem is Django, but I haven't been able to figure out why. Django is not printing or loggin any error!

Thanks for any help!

You could try setting the `X-Accel-Redirect` header and let nginx handle the file. That wouldn't be a bad idea either way, since this will currently hog up a Django thread for the full duration of the download. — knbk, Sep 05 '15 at 09:15
So when nginx serve the files it doesn't block his threads? I actually don't know how many threads has nginx because is `systemd` the one who is managing, but django is being managed by `gunicorn` and it has 7 threads — , Sep 05 '15 at 09:18
It won't block a gunicorn thread after Django has sent the header back to nginx. Threads in nginx should be less of a problem, though I'm not sure on the details. — knbk, Sep 05 '15 at 09:26
It seems that the `X-Accel-Redirect` is really what I need, but just in case some one else have this problem too, I found that my downloads/uploads was truncated by `gunicorn`, by adding the `--timeout` arg the problem was fixed. (But of-corse the threads are blodked as @knbk said it) — , Sep 05 '15 at 11:00
WHy on earth do you want to drag django into the picture. Nginx is perfect for handling large files. Just let ngix handle it — e4c5, Sep 05 '15 at 11:07
@e4c5 because I'm a newbie! I made my server/website by reading small tutorials and using my python experience. I still have a lot to learn and docs to read — , Sep 05 '15 at 12:01

e4c5 · Accepted Answer · 2016-07-04T15:31:12.757

Don't use django to deliver static content, specially not when it's static content that's as large as this. Nginx is ideal for delivering them. All you need to do is to create a mapping such as this in your nginx configuration file:

location /static/ {
     try_files $uri =404 ;
     root /var/www/myapp/;
     gzip on;
     gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;

}

With /var/www/myapp/ being the top level folder for your django app. Inside that you will have a folder named static/ into which you need to collect all your static files with the django manage.py 's collectstatic command.

Of course you are free to rename these folders anyway you like and to use a different file structure all together. More about how to configure nginx for static content at this link: http://nginx.org/en/docs/beginners_guide.html#static

score 0 · Answer 2 · answered Jun 06 '19 at 23:57

I ran into a similar problem which was visible in the nginx error log files by lines like this:

<TIMESTAMP> [error] 1221#1221: *913310 upstream prematurely closed connection 
while reading upstream, client: <IP>, server: <IP>, request: "GET <URL> HTTP/1.1",
upstream: "http://unix:<LOCAL_DJANGO_APP_DIR_PATH>/run/gunicorn.sock:
<REL_PATH_LOCAL_FILE_TO_BE_DOWNLOADED>", host: "<URL>", referrer: "<URL>/<PAGE>"

This is caused by the --timeout setting in the file

<LOCAL_DJANGO_APP_DIR_PATH>/bin/gunicorn_start 
(found at "command:" in /etc/supervisor/conf.d/<APPNAME>.conf)

In the gunicorn_start file change this line:

exec /usr/local/bin/gunicorn [...] \
--timeout <OLD_TIMEOUT> \
[...]

This was set to 300 and I had to change it to 1280 (this is in seconds!). Transfers of ~5GB are easily handled this way without RAM issues using

django.views.static.serve(request, <LOCAL_FILE_NAME>, <LOCAL_FILE_DIR>

NGinx & Django, serving large files (3gb+)

2 Answers2

Linked