0

Goal

I am making my own online server and recently added a feature enabling me to upload files and see the progress. I am building everything with Flask, Gunicorn and Nginx, pure CSS & JS.

Problem

However, when I try to upload large files (i.e. takes more than 30s to process), it freezes at 30 seconds of upload, then unfreezes at around 1min and freezes again 5 seconds later until 3.5 min where I get net::ERR_CONNECTION_RESET. I just don't know how to fix that and I've been working on it for hours now.

What I've tried

  • Using gevent workers for Gunicorn. I tried everything below with and without gevent workers.
  • Running gunicorn with --log-level debug --keep-alive 30 --timeout 300 --graceful-timeout 300. Nothing changed. Nothing gets shown in debug when trying to upload.
  • I added proxy_connect_timeout 300s;, proxy_read_timeout 300s; and client_max_body_size 100G; in the http section of my nginx.conf file just in case.

I also read all these posts: 1, 2, 3, 4, 5 and more, but none of these fixes worked. I'm helpless!

Relevant code if needed

Upload function:

function uploadFile(path) {
    let formdata = new FormData();
    let req = new XMLHttpRequest();
    let filesize = document.getElementById("fileInput").files[0].size;

    formdata.append("fileinput", document.getElementById("fileInput").files[0]);

    req.upload.addEventListener('progress', (uploadEvent) => {
        let progressElem = document.getElementById('progressIndicator');

        if (uploadEvent.loaded < filesize) {
            progressElem.innerHTML = (100 * uploadEvent.loaded / filesize).toFixed(2) + '%';
        } else {
            progressElem.style.color = "green";
            progressElem.innerHTML = "✅ File uploaded";
        }
    });

    req.open("POST", path);
    req.send(formdata);
}

views.py:

@app.route("/files/<path:path>", methods=["POST", "GET"]):
def files(path):
    if request.method == "POST":
        if not utils.is_uploader(str(session.get("session")), sessions) or os.path.isfile(BASE_DIR + path):
            return '', 403

        if not os.path.exists(BASE_DIR + path):
            return '', 404

        file = request.files.get("fileinput")
        if not file: return redirect(url_for("/files" + path)), 400

        secured_filename = secure_filename(file.filename or '')
        if secured_filename == '':
            return redirect(url_for("/files" + path)), 400

        file.save(os.path.join(BASE_DIR + path, secured_filename))

        return '', 204

    # rest not included
Biskweet
  • 130
  • 1
  • 9
  • Just a random comment; if you are not afraid to debug - I'd personally start be eliminating nginx from front of gunicorn and let gunicorn serve also the static assets so that you can atleast narrow things a bit .. does gunicorn have setting for timeout ? There was this https://stackoverflow.com/questions/43868863/server-fails-to-upload-large-files-with-gunicorn which could be related – rasjani Jan 23 '23 at 18:51
  • @rasjani yes gunicorn has a timeout option, which I tried with `--timeout 300`. No success. I really don't know what to do now. However, bypassing nginx and trying to serve directly on localhost seems a good idea; I doubt the problem comes from nginx though. – Biskweet Jan 23 '23 at 21:21
  • i wouldn't rule out nginx with default settings out yet.. I have faint recollection i've had somewhat similar issues with $httpd server proxy passing stuff to actual service. – rasjani Jan 23 '23 at 22:17

0 Answers0