Django+gunicorn+nginx upload large file 502 error

Question

Problem

Uploading 1-2mb files works fine. When I attempt to upload 16mb file, i get 502 error after several seconds

More detalied:

1. I click "Upload"
2. Google Chrome uploads file (upload status is changing from 0% to 100% in left bottom corner)
3. Status changes to "Waiting for HOST", where HOST is my site hostname
4. After a half of minute server returns "502 Bad Gateway"

My view:

def upload(request):
    if request.method == 'POST':
        f = File(data=request.FILES['file'])
        f.save()
        return redirect(reverse(display),  f.id)
    else:
        return render('filehosting_upload.html', request)

render(template, request [,data]) is my own shorthand that deals with some ajax stuff;

The filehosting_upload.html:

{% extends "base.html" %}
{% block content %}
    <h2>File upload</h2>
    <form action="{% url nexus.filehosting.views.upload %}" method="post" enctype="multipart/form-data">
        {% csrf_token %}
        <input type="file" name="file">
        <button type="submit" class="btn">Upload</button>
    </form>
{% endblock %}

Logs & specs

There are nothing informative in logs i can find.

Versions:

• Django==1.4.2
• Nginx==1.2.1
• gunicorn==0.17.2

Command line parameters

command=/var/www/ernado/data/envs/PROJECT_NAME/bin/gunicorn -b localhost:8801 -w 4 PROJECT_NAME:application

Nginx configuration for related location:

   location /files/upload {
    client_max_body_size 100m;
    proxy_pass http://HOST;
    proxy_connect_timeout 300s;
    proxy_read_timeout 300s;
   }

Nginx log entry (changed MY_IP and HOST)

2013/03/23 19:31:06 [error] 12701#0: *88 upstream prematurely closed connection while reading response header from upstream, client: MY_IP, server: HOST, request: "POST /files/upload HTTP/1.1", upstream: "http://127.0.0.1:8801/files/upload", host: "HOST", referrer: "http://HOST/files/upload"

Django log

2013-03-23 19:31:06 [12634] [CRITICAL] WORKER TIMEOUT (pid:12829)
2013-03-23 19:31:06 [12634] [CRITICAL] WORKER TIMEOUT (pid:12829)
2013-03-23 19:31:06 [13854] [INFO] Booting worker with pid: 13854

Question(s)

1. how to fix that?
2. is it possible to fix that without nginx upload module?

Update 1 Tried suggested config

 gunicorn --workers=3 --worker-class=tornado  --timeout=90 --graceful-timeout=10 --log-level=DEBUG --bind localhost:8801 --debug

Works fine for me now.

Answer 1

I run my gunicorn with that parameters, try :

python manage.py run_gunicorn --workers=3 --worker-class=tornado  --timeout=90 --graceful-timeout=10 --log-level=DEBUG --bind 127.0.0.1:8151 --debug

or if you run differently, you may run with that options

Answer 2

For large files handling you should use a worker-class. Also I had some trouble using gevent in python 3.7, better to use 3.6.

Django, Python 3.6 example:

Install:

pip install gevent

Run

gunicorn --chdir myApp myApp.wsgi --workers 4 --worker-class=gevent --bind 0.0.0.0:80 --timeout=90 --graceful-timeout=10

Answer 3

You need to used an other worker type class an async one like gevent or tornado see this for more explanation : First explantion :

You may also want to install Eventlet or Gevent if you expect that your application code may need to pause for extended periods of time during request processing

Second one :

The default synchronous workers assume that your application is resource bound in terms of CPU and network bandwidth. Generally this means that your application shouldn’t do anything that takes an undefined amount of time. For instance, a request to the internet meets this criteria. At some point the external network will fail in such a way that clients will pile up on your servers.