How to force http.client to send chunked-encoding HTTP body in python?

Question

I want to send chunked HTTP body to test my own HTTP server. So I wrote this python code:

import http.client

body = 'Hello World!' * 80

conn = http.client.HTTPConnection("some.domain.com")
url = "/some_path?arg=true_arg"

conn.request("POST", url, body, {"Transfer-Encoding":"chunked"})

resp = conn.getresponse()
print(resp.status, resp.reason)

I expect the HTTP request's body is transferrd chunked, but I capture the network package with Wireshark, the HTTP request's body is not transferred chunked.

How to transfer chunked body by http.client lib in python?

score 8 · Accepted Answer · answered Feb 11 '12 at 07:23

OK, I get it.

First, write my own chunked encode function.

Then use putrequest(), putheader(), endheaders() and send() instead of request()

import http.client

def chunk_data(data, chunk_size):
    dl = len(data)
    ret = ""
    for i in range(dl // chunk_size):
        ret += "%s\r\n" % (hex(chunk_size)[2:])
        ret += "%s\r\n\r\n" % (data[i * chunk_size : (i + 1) * chunk_size])

    if len(data) % chunk_size != 0:
        ret += "%s\r\n" % (hex(len(data) % chunk_size)[2:])
        ret += "%s\r\n" % (data[-(len(data) % chunk_size):])

    ret += "0\r\n\r\n"
    return ret


conn = http.client.HTTPConnection(host)
url = "/some_path"
conn.putrequest('POST', url)
conn.putheader('Transfer-Encoding', 'chunked')
conn.endheaders()
conn.send(chunk_data(body, size_per_chunk).encode('utf-8'))

resp = conn.getresponse()
print(resp.status, resp.reason)
conn.close()

works better for me whith only one separator in the second line inside the for of chunk_data (ie: ret += "%s\r\n" % (data[i * chunk_size : (i + 1) * chunk_size])) — PW., Dec 19 '12 at 17:14

score 2 · Answer 2 · edited May 23 '17 at 10:26

I'd suggest that if you already know the size of your data like in the answer given you could just set the Content-Length and send it all back in one hit, which is kind of what you're doing with the single call to conn.send anyway.

Chunked transfer encoding is most useful when you don't know how big the data is e.g. dynamically generated content. I've modified your code to illustrate:

import httplib

def write_chunk(conn, data):
    conn.send("%s\r\n" % hex(len(data))[2:])
    conn.send("%s\r\n" % data)

def dynamically_generate_data():
    for i in range(80):
        yield "hello world"

conn = httplib.HTTPConnection("localhost")
url = "/some_path"
conn.putrequest('POST', url)
conn.putheader('Transfer-Encoding', 'chunked')
conn.endheaders()

for new_chunk in dynamically_generate_data():
    write_chunk(conn, new_chunk)
conn.send('0\r\n')

resp = conn.getresponse()
print(resp.status, resp.reason)
conn.close()

How to force http.client to send chunked-encoding HTTP body in python?

2 Answers2

Linked