92

In the HTTP protocol you can send many requests in one socket using keep-alive and then receive the response from server at once, so that will significantly speed up whole process. Is there any way to do this in python requests lib? Or are there any other ways to speed this up that well using requests lib?

PaulOverflow
  • 1,091
  • 1
  • 10
  • 12

1 Answers1

162

Yes, there is. Use requests.Session and it will do keep-alive by default.

I guess I should include a quick example:

import logging
import requests

logging.basicConfig(level=logging.DEBUG)
s = requests.Session()
s.get('http://httpbin.org/cookies/set/sessioncookie/123456789')
s.get('http://httpbin.org/cookies/set/anothercookie/123456789')
r = s.get("http://httpbin.org/cookies")
print(r.text)

You will note that these log message occur

INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): httpbin.org
DEBUG:requests.packages.urllib3.connectionpool:"GET /cookies/set/sessioncookie/123456789 HTTP/1.1" 302 223
DEBUG:requests.packages.urllib3.connectionpool:"GET /cookies HTTP/1.1" 200 55
DEBUG:requests.packages.urllib3.connectionpool:"GET /cookies/set/anothercookie/123456789 HTTP/1.1" 302 223
DEBUG:requests.packages.urllib3.connectionpool:"GET /cookies HTTP/1.1" 200 90
DEBUG:requests.packages.urllib3.connectionpool:"GET /cookies HTTP/1.1" 200 90

If you wait a little while, and repeat the last get call

INFO:requests.packages.urllib3.connectionpool:Resetting dropped connection: httpbin.org
DEBUG:requests.packages.urllib3.connectionpool:"GET /cookies HTTP/1.1" 200 90

Note that it resets the dropped connection, i.e. reestablishing the connection to the server to make the new request.

metatoaster
  • 17,419
  • 5
  • 55
  • 66
  • 1
    I assumed `requests` was keeping the session alive, but in fact it was often not the case, and explicitly defining the session has greatly helped. – philshem Oct 01 '14 at 14:15
  • does proxies work with session keep-alive? I noticed that it did a resetting dropped connection after the first connection – chrizonline Feb 27 '15 at 03:37
  • It depends on how the proxy is configured. The proxy may be configured to ignore keep-alive as it may not want to track thousands of these persistent connections, or it might just silently drop them. – metatoaster Feb 27 '15 at 03:40
  • I want to use `requests` with `https`. So if I keep the session alive, should I skip verify for every `get` call(`requests.get(https://:/route, verfiy=True)`) which means I may use `verify=False`? Correct me if I am wrong. If I am correct, how do I know if the session is still alive? In other words, how long do I send the `get` requests with `verify=False`? – phanny Mar 05 '18 at 08:02
  • 3
    @phanny There is no automatic keep-alive with `requests.get`; use `requests.Session` as answered here with the `verify` set to `True`; e.g. `session = requests.Session(verify=True)`, and use `session.get(...)` – metatoaster Mar 05 '18 at 08:36
  • 6
    Thanks for the information @metatoaster. By the way, I think there is a little correction in the syntax. `session = requests.Session(verify=True)` was throwing me the error `TypeError: __init__() got an unexpected keyword argument 'verify'`. `session = requests.Session()` and then `session.verify = True` worked for me. – phanny Mar 05 '18 at 09:58
  • The link to the documentation is outdated. You can find it here: [keep-alive docs](https://requests.readthedocs.io/en/latest/user/advanced/?highlight=keep%20alive#keep-alive). Sadly I could not edit the answer directly due to the queue being full. – Christoph H. Jul 27 '22 at 09:59