Python3 requests.get is too slow

Question

EDIT Adding info:

requests version: 2.21.0

Server info: a Windows python implementation which includes 10 instances of threading.Thread, each creating HTTPServer with a handler based on BaseHTTPRequestHandler. My do_GET looks like this:

def do_GET(self):
    rc = 'some response'
    self.send_response(200)
    self.send_header('Content-type', 'text/html')
    self.send_header('Access-Control-Allow-Origin', '*')
    self.end_headers()
    self.wfile.write(rc.encode('utf-8'))

I'm getting a strange behaviour.

Using the curl command line, the GET command is finished quickly:

curl "http://localhost:3020/pbio/button2?cmd=uz-crosslink-leds&g1=0&g2=0&g3=0&g4=1&tmr=1"

However, using requests.get() of python takes too much time. I was isolated it up to

python -c "import requests; requests.get('http://localhost:3020/pbio/button2?cmd=uz-crosslink-leds&g1=0&g2=0&g3=0&g4=1&tmr=1')"

I scanned through many other questions here and have tried many things, without success.

Here are some of my findings:

If I'm adding timeout=0.2, the call is ending quickly without any error.
However, adding timeout=5 or timeout=(5,5)` doesn't make it take longer. It always seem to be waiting a full one second before returning with results.
Working with a session wrapper, and cancelling keep-alive, didn't improve. I mean for this:

with requests.Session() as session:
    session.headers.update({'Connection': 'close'})
    url = "http://localhost:3020/pbio/button2?cmd=uz-crosslink-leds&g1=0&g2=0&g3=0&g4=%d&tmr=0" % i
    session.get(url, timeout=2)

Enabling full debug, I'm getting the following output:

url=http://localhost:3020/pbio/button2?cmd=uz-crosslink-leds&g1=0&g2=0&g3=0&g4=1&tmr=0
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): localhost:3020
send: b'GET /pbio/button2?cmd=uz-crosslink-leds&g1=0&g2=0&g3=0&g4=1&tmr=0 HTTP/1.1\r\nHost: localhost:3020\r\nUser-Agent: python-requests/2.21.0\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: close\r\n\r\n'
reply: 'HTTP/1.0 200 OK\r\n'
header: Server: BaseHTTP/0.6 Python/3.7.2
header: Date: Wed, 01 May 2019 15:28:29 GMT
header: Content-type: text/html
header: Access-Control-Allow-Origin: *
DEBUG:urllib3.connectionpool:http://localhost:3020 "GET /pbio/button2?cmd=uz-crosslink-leds&g1=0&g2=0&g3=0&g4=1&tmr=0 HTTP/1.1" 200 None
url=http://localhost:3020/pbio/powermtr?cmd=read-power-density
DEBUG:urllib3.connectionpool:Resetting dropped connection: localhost
slight pause here
send: b'GET /pbio/powermtr?cmd=read-power-density HTTP/1.1\r\nHost: localhost:3020\r\nUser-Agent: python-requests/2.21.0\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: close\r\n\r\n'
reply: 'HTTP/1.0 200 OK\r\n'
header: Server: BaseHTTP/0.6 Python/3.7.2
header: Date: Wed, 01 May 2019 15:28:30 GMT
header: Content-type: text/html
header: Access-Control-Allow-Origin: *
DEBUG:urllib3.connectionpool:http://localhost:3020 "GET /pbio/powermtr?cmd=read-power-density HTTP/1.1" 200 None
6.710,i=4
url=http://localhost:3020/pbio/button2?cmd=uz-crosslink-leds&g1=0&g2=0&g3=0&g4=4&tmr=0
DEBUG:urllib3.connectionpool:Resetting dropped connection: localhost
slight pause here
...

What server is this connecting to? Have you got any performance metrics for that setup? If you replaced `http://localhost:3020/pbio/button2` with `http://httpbin.org/get` do you see the same differences in performance? — Martijn Pieters, May 08 '19 at 12:41
I might overlook something obvious here but your python debug does **two** get requests when your curl would obviously only make one. For further troubleshooting, you can also do `curl -v "http://localhost:3020/pbio/button2?cmd=uz-crosslink-leds&g1=0&g2=0&g3=0&g4=1&tmr=1"` then cross check the get request of the curl and the get request of python matches — β.εηοιτ.βε, May 09 '19 at 21:46
Another point of note: `curl` can't keep connections open, because it makes one request per process. So a badly behaving server that keeps connections open then drops them after a timeout will not faze `curl` as it has already closed the connection out of necessity. — Martijn Pieters, May 11 '19 at 19:44
Finally: `Connection: close` is a HTTP/1.1 header, but the server you are connecting to states it implements HTTP/1.0. It is the server's responsibility to close the socket properly, clearly it is not doing so here. — Martijn Pieters, May 11 '19 at 19:53
Thank you all for your comments. I did provide some more info. The server is a standard Python server and I didn't find anything I need to do in order to close the request. My `do_GET()` is ending with `self.wfile.write(rc.encode('utf-8'))` — ishahak, May 12 '19 at 15:13
If at all possible, maybe try a more modern server library like [Flask](http://flask.pocoo.org/) **(Hello World is 6 lines)** — Eric Reed, May 12 '19 at 17:47
I think that you should write `session.verify = True` to ignore the session verification if it's a live or not — Neji Soltani, May 13 '19 at 10:45
Hi @NejiSoltani, I was suspecting that myself. Still no change. — ishahak, May 13 '19 at 13:42

score -1 · Answer 1 · answered May 09 '19 at 21:31

From the docs:

timeout is not a time limit on the entire response download; rather, an exception is raised if the server has not issued a response for timeout seconds (more precisely, if no bytes have been received on the underlying socket for timeout seconds). If no timeout is specified explicitly, requests do not time out.

score -1 · Accepted Answer · answered Jun 01 '22 at 14:26

It took me 3 years to find an answer. I still do not understand why, but at least I can suggest a working solution.

According to these docs, the timeout can be specified as a tuple, like this:

(timeout for connection, timeout for interval without data)

Although I do not understand why requests is waiting for [timeout] before issuing the connection, I can tell it to wait very little for the connection, and specify another timeout for the data.

So what I'm doing now, is giving a timeout of let's say (0.01, 4). Now the connection is immediate, and if the data has a deadtime of 4 seconds, it will generate a timeout exception.

Some interesting reading can be found here.

Hoping this info will help others!

Python3 requests.get is too slow

2 Answers2