Content-Length returned in headers for chunked transfer encoding with nginx, uwsgi and flask

Question

If I use chunked transfer encoding with nginx, uwsgi and flask I always get Content-Length in headers together with Transfer-Encoding: chunked. However, HTTP 1.1 prohibits this behaviour. I have tried to configure nginx and uwsgi to achieve desired behaviour (no Content-Length in headers when Transfer-Encoding: chunked) but without success. Firstly, there is my server and client code:

Server code (server.py):

from flask import Flask
from flask import request

application = Flask(__name__)

@application.route('/', methods=['PUT'])
def hello():
    print(request.headers)
    print(request.environ.get('SERVER_PROTOCOL'))
    return "Hello World!"

if __name__ == "__main__":
    application.run(host='0.0.0.0')

Client code (client.py):

import requests

def get_data():
    yield b'This is test file.'
    yield b'This is test file.'

r = requests.request(
    method='PUT',
    url='http://127.0.0.1:5000/',
    data=get_data(),
    headers={
        'Content-type': 'text/plain',
        'X-Accel-Buffering': 'no',
    }
)
print('Response: ', r.text)

If I run server and try to connect to server with client I get the following output. Server output:

* Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
Host: 127.0.0.1:5000
User-Agent: python-requests/2.18.4
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive
Content-Type: text/plain
X-Accel-Buffering: no
Transfer-Encoding: chunked


HTTP/1.1
[pid: 18455|app: 0|req: 1/1] 127.0.0.1 () {34 vars in 412 bytes} [Wed Jan 17 08:24:53 2018] PUT / => generated 12 bytes in 0 msecs (HTTP/1.1 200) 2 headers in 79 bytes (1 switches on core 0)

Client output:

Response:  Hello World!

For now, everything seems alright. In headers, we have Transfer-Encoding without Content-Length. Now, I try to incorporate uwsgi (uwsgi.py):

from server import application

if __name__ == "__main__":
    application.run()

I run the following command:

$ uwsgi --http-socket localhost:5000 -w wsgi

The output is the same as in the previous attempt. Therefore, still as expected. Now, I will try to deploy nginx. My uwsgi configuration (uwsgi.ini):

[uwsgi]
module = wsgi

master = true
processes = 5

socket = /tmp/flask.sock
chmod-socket = 777
vacuum = true

die-on-term = true

My nginx configuration (/etc/nginx/nginx.conf):

worker_processes  1;
events {
    worker_connections  1024;
}
http {
    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;
    keepalive_timeout  65;

    server {
        listen 5000;
        server_name 127.0.0.1;

        location / {
            include uwsgi_params;
            uwsgi_pass unix:/tmp/flask.sock;
            proxy_request_buffering off;
            proxy_buffering off;
            proxy_http_version 1.1;
            chunked_transfer_encoding on;
        }
        proxy_request_buffering off;
        proxy_buffering off;
        proxy_http_version 1.1;
        chunked_transfer_encoding on;
    }
}

I start nginx and then I run:

$ uwsgi --ini uwsgi.ini --wsgi-manage-chunked-input --http-raw-body --http-auto-chunked --http-chunked-input

Now, the output contains Content-Length:

Content-Type: text/plain
Content-Length: 36
Host: 127.0.0.1:5000
User-Agent: python-requests/2.18.4
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive
X-Accel-Buffering: no
Transfer-Encoding: chunked

HTTP/1.1
[pid: 20220|app: 0|req: 1/1] 127.0.0.1 () {42 vars in 525 bytes} [Wed Jan 17 08:31:39 2018] PUT / => generated 12 bytes in 1 msecs (HTTP/1.1 200) 2 headers in 79 bytes (1 switches on core 0)

I try different settings for nginx: proxy_request_buffering, proxy_buffering, proxy_http_version and chunked_transfer_encoding in the server and location context without success. I added X-Accel-Buffering: no to headers but it did not solve the problem. Also, I tried different options for uwsgi: wsgi-manage-chunked-input, http-raw-body, http-auto-chunked, http-chunked-input without achieving desired behaviour (Content-Length is still present in headers with Transfer-Encoding: chunked).

I use the following versions of flask, uwsgi and nginx:

Flask==0.12.2
uWSGI===2.1-dev-f74553db
nginx 1.12.2-2

Any idea what can be wrong? Thanks.

not sure, but maybe? : https://stackoverflow.com/a/27043532/2026508 and set the `stream=True` on the `Request`? — jmunsch, Jan 17 '18 at 09:28
@jmunsch The referenced post deals with a different problem. As for `stream=True`, this does not have any impact on the request. It controls the obtaining of the response. — sopticek, Jan 21 '18 at 10:24

kidokim · Answer 1 · 2018-06-02T17:15:21.187

I suffered same problem. My environment was uWSGI 2.0.17, with Nginx and Flask 1.0. When the client sent a request with Transfer-Encoding: chunked, the nginx added the Content-Length. (Though this is prohibited by HTTP/1.1 protocol.)

My conclusion was uWSGI does not support request with chunked Tranfer-Encoding. My flask app didn't get the request body of POST method when the request header have Transfer-Encoding only and also have both Transfer-Encoding and Content-Length.

Waitress, the wsgi server for python 2 and 3, solved this problem. If waitress receives the request with Transfer-Encoding header, he ignores it and set the right Content-Length. (refer to https://github.com/Pylons/waitress/blob/c18aa5e24e8d96bb64c9454a626147f37a23f0f0/waitress/parser.py#L154)

The flask official document also recommends the waitree to run a production flask server. (http://flask.pocoo.org/docs/1.0/tutorial/deploy/)

Content-Length returned in headers for chunked transfer encoding with nginx, uwsgi and flask

1 Answers1