nginx - connection timed out while reading upstream

Question

I have a flask server with and endpoint that processes some uploaded .csv files and returns a .zip (in a JSON reponse, as a base64 string) This process can take up to 90 seconds

I've been setting it up for production using gunicorn and nginx and I'm testing the endpoint with smaller .csv s. They get processed fine and in a couple seconds I get the "got blob" log. But nginx doesn't return it to the client and finally it times out. I set up a longer fail-timeout of 10 minutes and the client WILL wait 10 minutes, then time out

the proxy read timeout offered as solution here is set to 3600s

Also the proxy connect timeout is set to 75s according to this

also the timeout for the gunicorn workers according to this

The error log says: "upstream timed out connection timed out while reading upstream" I also see examples of nginx receiving an OPTIONS request and immediately after the POST request (some CORS weirdness from the client) where nginx passes the OPTIONS request but fails to pass the POST request to gunicorn despite nginx having received it

Question: What am I doing wrong here? Many thanks

http {
    upstream flask  { 
        server 127.0.0.1:5050 fail_timeout=600; 
    }
    # error log
    # 2022/08/18 14:49:11 [error] 1028#1028: *39 upstream timed out (110: Connection timed out) while reading upstream, ...
    # ...
    server {
        # ...
        location /api/ {
            proxy_pass http://flask/;
            proxy_read_timeout 3600;
            proxy_connect_timeout 75s;
            # ...
        }
    # ...
    }
}

# wsgi.py
from main import app
if __name__ == '__main__':
    app.run()

# flask endpoint
@app.route("/process-csv", methods=['POST'])
def process_csv():
    def wrapped_run_func():
        return blob, export_filename
    # ...
    try:
        blob, export_filename = wrapped_run_func()
        b64_file = base64.b64encode(blob.getvalue()).decode()
        ret = jsonify(file=b64_file, filename=export_filename)
        # return Response(response=ret, status=200, mimetype="application/json")
        print("got blob")
        return ret
    except Exception as e:
        app.logger.exception(f"0: Error processing file: {export_filename}")
        return  Response("Internal server error", status=500)

ps. getting this error from stackoverflow "Your post appears to contain code that is not properly formatted as code. Please indent all code by 4 spaces using the code toolbar button or the CTRL+K keyboard shortcut. For more editing help, click the [?] toolbar icon." for having perfectly well formatted code with language syntax, I'm sorry that I had to post it ugly

score 0 · Answer 1 · answered Aug 24 '22 at 09:19

Sadly I got no response

See last lines for the "solution" finally implemented

CAUSE OF ERROR: I believe the problem is that I'm hosting the Nginx server on wsl1

I tried updating to wsl2 and see if that fixed it but I need to enable some kind of "nested virtualization", as the wsl1 is running already on a VM.

Through conf changes I got it to the point where no error is logged, gunicorn return the file then it just stays in the ether. Nginx never gets/sends the response

"SOLUTION": I ended up changing the code for the client, the server and the nginx.conf file:

the server saves the resulting file and only returns the file name

the client inserts the filename into an href that then displays a link

on click a request is sent to nginx which in turn just sends the file from a static folder, leaving gunicorn alone

I guess this is the optimal way to do it anyway, though it still bugs me I couldn't (for sure) find the reason of the error

nginx - connection timed out while reading upstream

1 Answers1