12

I have a view in a Django application that take as input a large CSV file. It cycle all the rows and insert datas on DB. Locally I have no problem but when I deploy my proj online the view give me back a timeout after some time. My webserver configuration is Nginx + Gunicorn. I tried to increase timeout on Gunicorn and proxy_connect_timeout, proxy_read_timeout on Nginx by writing a large number (120000) and now it's better, I get timeout after about 90 seconds instead of 30 (default for Gunicorn) but still is not what I expected and it's not enough to finish my job. Also I don't like so much this approach. I don't want infinite timeout for every request. What's the best approach to deal with long request and not have timeout? Maybe by answering to user with a message and then run the job in background? Any suggestion? Thanks.

Bip Lob
  • 485
  • 2
  • 6
  • 15
Alessio
  • 143
  • 1
  • 2
  • 7
  • 1
    Don't try to increase timeout. A view should respond quickly when it is invoked. If you need to perform a long running task, you should look around asynchronous job execution. Celery is best for this, but you can also run your job in a different thread using python threading. See https://docs.python.org/3.6/library/threading.html – Antwane Aug 02 '17 at 11:52
  • 1
    Yes you're right, increasing timeout is not the solution that's why I asked..thanks for your answer. – Alessio Aug 02 '17 at 13:35
  • For jobs like that, you should use task queue such as celery+rabbitMQ. –  Aug 02 '17 at 11:39

2 Answers2

7

Using Celery With Django for Background Task Processing that means process the CSV file with celery async task.

OR

As a quick hack, if you don't want to use celery; use multi-threading to process the CSV and save the result of the processing in DB or file and server the result from DB or file.

Note: Never Process big files on main thread; always try to use a different server to process the big files.If different server is not possible then try to process it in background task

Many solutions can be found on this StackOverflow link

Himanshu dua
  • 2,496
  • 1
  • 20
  • 27
  • I thought to use Celery but, as it's just a backend option for admin and maybe it's not used so much, I prefer to don't add Celery and RabbitMQ/Redis to my stack so I was searching if there is an easier solution...what about the second part of your answer? I didn't get it, could you explain it better please? – Alessio Aug 02 '17 at 11:47
  • 1
    I used Celery with Redis and I solved my problems, thanks. – Alessio Aug 04 '17 at 13:33
6
    I spend more time to increase the request timeout. And Finally I got the soluton
    For **Django Rest API ( Nginx + supervisor + Gunicorn + AWS )**

     1. Add timeout in the gunicorn(Supervisor config)

        [program:gunicorn]
         command = <env_path>/env/bin/gunicorn --pythonpath=<python_path> --name=<nginx_name> --bind unix:/tmp/myproject.sock --workers 3 --graceful-timeout=900 --timeout=900 <project_path>.wsgi:application
        directory=<project_path>
        autostart=true
        autorestart=true
        stopsignal=TERM
        stdout_logfile=/var/log/gunicorn.stdout.log
        stdout_logfile_maxbytes=1MB
        stdout_logfile_backups=5
        stderr_logfile=/var/log/gunicorn.stderr.log
        stderr_logfile_maxbytes=1MB
        stderr_logfile_backups=5

     2. Add proxy on nginx file

     proxy_connect_timeout       900;
     proxy_send_timeout          900;
     proxy_read_timeout          900;
     send_timeout                900;

     3. Changes in Load Balancer ( If you configure  -> in EC2 instance)
 [Load Balancer][1]


  [1]: https://i.stack.imgur.com/uSUrK.png
Rajan Mandanka
  • 2,003
  • 21
  • 13
  • You can find Load Balancer screeshot here for reference https://i.stack.imgur.com/uSUrK.png – Rajan Mandanka Jul 02 '18 at 06:02
  • This is what helped me ! Thank you! The problem was i needed to add "--graceful-timeout=900 --timeout=900" to my gunicorn file and provided proxy settings to nginx. Now it doesnt timeout when loading alot of data. – Paul Grei Dec 23 '21 at 14:33