Python httplib ResponseNotReady

Question

I'm writing a REST client for elgg using python, and even when the request succeeds, I get this in response:

Traceback (most recent call last):
  File "testclient.py", line 94, in <module>
    result = sendMessage(token, h1)
  File "testclient.py", line 46, in sendMessage
    res = h1.getresponse().read()
  File "C:\Python25\lib\httplib.py", line 918, in getresponse
    raise ResponseNotReady()
httplib.ResponseNotReady

Looking at the header, I see ('content-length', '5749'), so I know there is a page there, but I can't use .read() to see it because the exception comes up. What does ResponseNotReady mean and why can't I see the content that was returned?

Indeed. Oddly, sometimes it works and sometimes it doesn't. I can't determine what behavior determines that though. — directedition, Jul 12 '10 at 21:39

score 81 · Answer 1 · edited Apr 04 '18 at 18:04

81

Previous answers are correct, but there's another case where you could get that exception:

Making multiple requests without reading any intermediate responses completely.

For instance:

conn.request('PUT',...)
conn.request('GET',...)
# will not work: raises ResponseNotReady

conn.request('PUT',...)
r = conn.getresponse()
r.read() # <-- that's the important call!
conn.request('GET',...)
r = conn.getresponse()
r.read() # <-- same thing

and so on.

edited Apr 04 '18 at 18:04

Keliomer Castillo

123
11

answered Jun 04 '12 at 04:03

Bokeh

1,013
7
9

1

Thats what I was needing .read() so I can reuse my connection. – Nathan Schwermann Feb 11 '14 at 19:54
1

is .read() expensive. What if I don't care about the response? Can I make it run faster ?> – Neil Jul 07 '16 at 06:58
6

@ronnefeldt, Python docs at https://docs.python.org/3/library/http.client.html has the following note: "Note that you must have read the whole response before you can send a new request to the server.". – mvsagar Jul 12 '16 at 06:15
If you hadn't said this I would have had a very upsetting next few hours maybe even days. THANK YOU – Daniel Cull May 29 '18 at 16:28

score 46 · Accepted Answer · answered Jul 12 '10 at 23:13

46

Make sure you don't reuse the same object from a previous connection. You will hit this once the server keep-alive ends and the socket closes.

answered Jul 12 '10 at 23:13

hcalves

2,268
1
21
17

2

Generally, as a rule, I don't bother trying to reuse HttpRequest objects unless I have a specific, performance-driven need to do so. Just single-shot 'em – rossipedia Aug 06 '10 at 20:31
In my case, adding `preload_content=False` solved the issue. Here is the snippet: `http = urllib3.PoolManager(threadsNo, maxsize=threadsNo, block=True); request = http.request('GET', queryUrl, preload_content=False)` It looks like `request.release_conn()` does not really release the previous connection object unless above parameter is passed to http.request. More details here: https://urllib3.readthedocs.org/en/latest/pools.html – Dex Sep 08 '15 at 07:06

score 2 · Answer 3 · answered May 01 '15 at 17:10

2

Additionally, errors like this can occur when the server sends a response without a Content-Length header, which will cripple the state of the HTTP client if Keep-Alive is used and another request is sent over the same socket.

answered May 01 '15 at 17:10

wouter bolsterlee

3,879
22
30

score 2 · Answer 4 · answered Aug 06 '10 at 20:29

I was running into this same exception today, using this code:

    conn = httplib.HTTPConnection(self._host, self._port)
    conn.putrequest('GET',
        '/retrieve?id={0}'.format(parsed_store_response['id']))
    retr_response = conn.getresponse()

I didn't notice that I was using putrequest rather than request; I was mixing my interfaces. ResponseNotReady is raised because I haven't actually sent the request yet.

score 0 · Answer 5 · answered Feb 06 '18 at 07:58

0

This can also occur if a firewall blocks the connection.

answered Feb 06 '18 at 07:58

Tommy

594
5
8

score 0 · Answer 6 · answered Mar 12 '19 at 04:43

Unable to add comment to @Bokeh 's answer; as I do not have the requisite reputation yet on this platform.

So, adding as answer: Bokeh's answer worked for me.

I was trying to pipeline multiple requests sequentially over the same connection object. For few of the responses I wanted to process the response later, hence missed to read the response.

From my experience, I second Bokeh's answer:

response.read() is a must after each request. Even if you wish to process response or not.

From my standpoint this question would have been incomplete without Bokeh's answer. Thanks @Bokeh

Python httplib ResponseNotReady

6 Answers6

Linked