10

I'm using the python requests module to do some testing against a site.

The requests module allows you to remove certain headers by passing in a dictionary with the keys set to None. For example

headers = {u'User-Agent': None}

will ensure that no user agent is sent with the request.

However, it seems that when I post data, requests will calculate the correct Content-Length for me even if I specify None, or an incorrect value. Eg.

headers = {u'Content-Length': u'999'}
headers = {u'Content-Length': None}

I check the response for the headers used in the request (response.request.headers) and I can see the Content-Length has been re-added with the correct value. So Far I cannot see any way to disable this behaviour

CaseInsensitiveDict({'Content-Length': '39', 'Content-Type': 'application/x-www-form-urlencoded', 'Accept-Encoding': 'gzip, deflate, compress', 'Accept': '*/*', 'User-Agent': 'python-requests/2.2.1 CPython/2.7.6 Linux/3.13.0-36-generic'})

I'd REALLY like to remain with the requests module to do this. Is this possible?

HoldOffHunger
  • 18,769
  • 10
  • 104
  • 133
joeButler
  • 1,643
  • 1
  • 20
  • 41

1 Answers1

20

You'll have to prepare the request manually, then remove the generated content-length header:

from requests import Request, Session

s = Session()
req = Request('POST', url, data=data)
prepped = req.prepare()
del prepped.headers['content-length']
response = s.send(prepped)

Do note that most compliant HTTP servers may then ignore your post body!

If you meant to use chunked transfer encoding (where you don't have to send a content length), then use a iterator for the data parameter. See Chunked-Encoded Requests in the documentation. No Content-Length header will be set in that case.

Martijn Pieters
  • 1,048,767
  • 296
  • 4,058
  • 3,343
  • FYI, Requests >= 2.0, as well as built-in `urllib.request` and `http.client` in Python >= 3.6, swtich to `Transfer-Encoding: chunked` automatically when `Content-Length` is not present. See https://github.com/psf/requests/commit/ef8563ab36c6b52834ee9c35f6f75a424cd9ceef and https://github.com/python/cpython/commit/3c0d0baf2badfad7deb346d1043f7d83bb92691f . I wasted quite a lot of time trying to submit a request without `Content-Length` before finding this out. – selurvedu Jul 29 '19 at 16:53
  • 1
    @selurvedu: the content header is not added when using an iterator for the data parameter, or if the body otherwise doesn't have a length. At that point `Transfer-Encoding: chunked` is used, which is what I refer to with my last paragraph. – Martijn Pieters Jul 29 '19 at 17:54
  • @ Martijn, sure, that's what I was saying. I missed the iterator part, but using an iterator implies that content length cannot be easily determined. – selurvedu Jul 30 '19 at 18:23
  • 1
    @selurvedu: yes, which is the point. If content length can easily be determined, you definitely should use a content length in your request. – Martijn Pieters Jul 31 '19 at 10:01