Python HTTP server that supports chunked encoding?

Question

I'm looking for a well-supported multithreaded Python HTTP server that supports chunked encoding replies. (I.e. "Transfer-Encoding: chunked" on responses). What's the best HTTP server base to start with for this purpose?

Answer 1

Twisted supports chunked transfer encoding (API link) (see also the API doc for HTTPChannel). There are a number of production-grade projects using Twisted (for example, Apple uses it for the iCalendar server in Mac OS X Server), so it's quite well supported and very robust.

Answer 2

Twisted supports chunked transfer and it does so transparently. i.e., if your request handler does not specify a response length, twisted will automatically switch to chunked transfer and it will generate one chunk per call to Request.write.

Answer 3

You can implement a simple chunked server using Python's HTTPServer, by adding this to your serve function:

    def _serve(self, res):
        response = next(res)

        content_type = 'application/json'

        self.send_response(200)
        self.send_header('Content-Type', content_type)
        self.send_header('Transfer-Encoding', 'chunked')
        self.end_headers()

        try:
            while True:
                # This line removed as suggested by @SergeyNudnov
                # r = response.encode('utf-8')
                l = len(r)
                self.wfile.write('{:X}\r\n{}\r\n'.format(l, r).encode('utf-8'))

                response = next(it)
        except StopIteration:
            pass

        self.wfile.write('0\r\n\r\n'.encode('utf-8'))

I would not recommend it for production use.

Answer 4

I managed to do it using Tornado:

#!/usr/bin/env python

import logging

import tornado.httpserver
import tornado.ioloop
import tornado.options
import tornado.web

from tornado.options import define, options

define("port", default=8080, help="run on the given port", type=int)

@tornado.web.stream_request_body
class MainHandler(tornado.web.RequestHandler):
    def post(self):
        print()
    def data_received(self, chunk):
        self.write(chunk)

        logging.info(chunk)

def main():
    tornado.options.parse_command_line()

    application = tornado.web.Application([
        (r"/", MainHandler),
    ])

    http_server = tornado.httpserver.HTTPServer(application)
    http_server.listen(options.port)

    tornado.ioloop.IOLoop.current().start()

if __name__ == "__main__":
    main()

Answer 5

The script below is a full working example. It could be used as a CGI script to stream data under Apache or IIS:

#!/usr/bin/env pythonw
import sys
import os
import time

# Minimal length of response to avoid its buffering by IIS+FastCGI
# This value was found by testing this script from a browser and
# ensuring that every event received separately and in full
response_padding = 284


def send_chunk(r):
    # Binary write into stdout
    os.write(1, "{:X}\r\n{}\r\n".format(len(r), r).encode('utf-8'))


class Unbuffered(object):
    """
    Stream wrapper to disable output buffering
    To be used in the CGI scripts
    https://stackoverflow.com/a/107717/9921853
    """
    def __init__(self, stream):
        self.stream = stream

    def write(self, data):
        self.stream.write(data)
        self.stream.flush()

    def writelines(self, lines):
        self.stream.writelines(lines)
        self.stream.flush()

    def __getattr__(self, attr):
        return getattr(self.stream, attr)


# Ensure stdout is unbuffered to avoid problems serving this CGI script on IIS
# Also web.config should have responseBufferLimit="0" applied to the CgiModule handler
sys.stdout = Unbuffered(sys.stdout)
print(
    "Transfer-Encoding: chunked\n"
    "Content-Type: text/event-stream; charset=utf-8\n"
)

# Fixing an issue, that IIS provides a wrong file descriptor for stdin, if no data passed to the POST request
sys.stdin = sys.stdin or open(os.devnull, 'r')

progress = 0

send_chunk(
    (
        "event: started\n"
        f"data: {progress}"
    ).ljust(response_padding) + "\n\n"
)

while progress < 5:
    time.sleep(2)
    progress += 1

    send_chunk(
        (
            "event: progress\n"
            f"data: {progress}"
        ).ljust(response_padding) + "\n\n"
    )

time.sleep(2)

send_chunk(
    (
        "event: completed\n"
        f"data: {progress+1}"
    ).ljust(response_padding) + "\n\n"
)

# To close stream
send_chunk('')

########################################################
# All Done

Answer 6

I am pretty sure that WSGI compliant servers should support that. Essentially, WSGI applications return iterable chunks, which the webserver returns. I don't have first hand experience with this, but here is a list of compliant servers.

I should think that it would be fairly easy to roll your own though, if WSGI servers dont meet what you are looking for, using the Python's builtin CGIHTTPServer. It is already multithreaded, so it would just be up to you to chunk the responses.