71

I was wondering, how do you close a connection with Requests (python-requests.org)?

With httplib it's HTTPConnection.close(), but how do I do the same with Requests?

Code:

r = requests.post("https://stream.twitter.com/1/statuses/filter.json", data={'track':toTrack}, auth=('username', 'passwd'))
for line in r.iter_lines():
    if line:
        self.mongo['db'].tweets.insert(json.loads(line))
rkta
  • 3,959
  • 7
  • 25
  • 37
  • I don't think Requests establishes a persistent connection... there's no connection to close. –  Apr 11 '12 at 23:39
  • @michael, you get a persistent connection pool for free unless you specify otherwise http://docs.python-requests.org/en/latest/user/advanced/#keep-alive – istruble Apr 11 '12 at 23:43
  • @istruble Then urllib3 handles cleaning up the connection pool? –  Apr 11 '12 at 23:46

8 Answers8

77

I think a more reliable way of closing a connection is to tell the sever explicitly to close it in a way compliant with HTTP specification:

HTTP/1.1 defines the "close" connection option for the sender to signal that the connection will be closed after completion of the response. For example,

   Connection: close

in either the request or the response header fields indicates that the connection SHOULD NOT be considered `persistent' (section 8.1) after the current request/response is complete.

The Connection: close header is added to the actual request:

r = requests.post(url=url, data=body, headers={'Connection':'close'})
WoJ
  • 27,165
  • 48
  • 180
  • 345
Oleg Gryb
  • 5,122
  • 1
  • 28
  • 40
  • What is 'body' in this case? just any { }? – Brad Jan 18 '15 at 22:04
  • 1
    @Brad It depends on content-type, e.g. if you want to send application/json, you can use data=json.dumps(payload), if you just want to send a common form-encoded payload, you can use data={"k1":"v1", ...}. See more details here: http://docs.python-requests.org/en/latest/user/quickstart/#more-complicated-post-requests – Oleg Gryb Jan 19 '15 at 17:00
  • 1
    That's being used [here](https://github.com/smilli/py-corenlp/blob/master/pycorenlp/corenlp.py#L29) but for some reason it appears that the connection is not really closed - I notices because my OS runs out of ports - which should not happen imho if that connection gets closed correctly. :/ – Stefan Falk Jan 08 '18 at 13:35
71

I came to this question looking to solve the "too many open files" error, but I am using requests.session() in my code. A few searches later and I came up with an answer on the Python Requests Documentation which suggests to use the with block so that the session is closed even if there are unhandled exceptions:

with requests.Session() as s:
    s.get('http://google.com')

If you're not using Session you can actually do the same thing: https://2.python-requests.org/en/master/user/advanced/#session-objects

with requests.get('http://httpbin.org/get', stream=True) as r:
    # Do something
stwhite
  • 3,156
  • 4
  • 37
  • 70
  • 5
    Am I supposed to downvote other comments in order to help this one to go up? – dmigo Dec 06 '18 at 13:10
  • @dmigo: No. Only downvote if an answer is wrong in some way, or otherwise not useful to the question asked. – Ethan Furman Jan 23 '20 at 20:22
  • Lets say i create a session using "with" in one function and return the session. Use the session in all my tests. How would i know if the session is terminated? – manoj prashant k Feb 10 '20 at 09:36
  • @manojprashantk The best answer I can give is to test it in your code. Likely the request will be closed once you're out of the `with` block but you should read about Context Managers as it may have more of an answer to your question: https://docs.python.org/2/reference/datamodel.html#context-managers – stwhite Feb 10 '20 at 19:33
  • Are you sure this works? requests.get() behind the scenes also uses the same construct, and the keep-alive is still working ... – Melardev Mar 29 '20 at 21:17
32

As discussed here, there really isn't such a thing as an HTTP connection and what httplib refers to as the HTTPConnection is really the underlying TCP connection which doesn't really know much about your requests at all. Requests abstracts that away and you won't ever see it.

The newest version of Requests does in fact keep the TCP connection alive after your request.. If you do want your TCP connections to close, you can just configure the requests to not use keep-alive.

s = requests.session()
s.config['keep_alive'] = False
Community
  • 1
  • 1
Felix Fung
  • 1,711
  • 19
  • 27
  • 15
    Alternatively `s = requests.session(config={'keep_alive': False})` – Piotr Dobrogost Apr 12 '12 at 09:16
  • 6
    OP was asking how to close a streaming response. I don't see how setting keep-alive on the session is going to help there. There are several recent tickets in requests and urllib3 around this issue: https://github.com/shazow/urllib3/pull/132, https://github.com/kennethreitz/requests/issues/1073, https://github.com/kennethreitz/requests/issues/1041 – btubbs Jan 11 '13 at 00:35
  • See comment here: https://github.com/shazow/urllib3/pull/132#issuecomment-11516729 – Felix Fung Jan 11 '13 at 23:14
  • 38
    Current `request` versions have *no support* for switching off Keep-Alive. – Martijn Pieters Apr 11 '14 at 21:13
  • 3
    As of the current release you can ensure streaming connections go back to the pool by reading Response.content (or Response.text which calls Response.content). But if you are hoarding Response objects without reading them you will not be releasing your connections back: [Documentation](http://docs.python-requests.org/en/latest/user/advanced/#keep-alive) – gss May 31 '16 at 16:36
  • 4
    TypeError: session() got an unexpected keyword argument 'config' –  Nov 24 '20 at 10:01
26

please use response.close() to close to avoid "too many open files" error

for example:

r = requests.post("https://stream.twitter.com/1/statuses/filter.json", data={'track':toTrack}, auth=('username', 'passwd'))
....
r.close()
JochenJung
  • 7,183
  • 12
  • 64
  • 113
hatcher
  • 361
  • 3
  • 4
12

On Requests 1.X, the connection is available on the response object:

r = requests.post("https://stream.twitter.com/1/statuses/filter.json",
                  data={'track': toTrack}, auth=('username', 'passwd'))

r.connection.close()
Wilfred Hughes
  • 29,846
  • 15
  • 139
  • 192
3

this works for me:

res = requests.get(<url>, timeout=10).content
requests.session().close()
user1990614
  • 41
  • 3
2

Based on the latest requests(2.25.1), the requests.<method> will close the connection by default

with sessions.Session() as session:
    return session.request(method=method, url=url, **kwargs)

https://github.com/psf/requests/blob/master/requests/api.py#L60

Thus, if you use the latest version of requests, it seems we don't need to close the connection by ourselves.

Also, if you need to send multiple times of requests with the same session, it's better to use requests.Session() instead of open/close the connection multiple times. EX:

with requests.Session() as s:
    r = s.get('https://example.org/1/')
    print(r.text)
    r = s.get('https://example.org/2/')
    print(r.text)
    r = s.get('https://example.org/3/')
    print(r.text)
user1995868
  • 233
  • 1
  • 2
  • 11
0

To remove the "keep-alive" header in requests, I just created it from the Request object and then send it with Session

headers = {
'Host' : '1.2.3.4',
'User-Agent' : 'Test client (x86_64-pc-linux-gnu 7.16.3)',
'Accept' : '*/*',
'Accept-Encoding' : 'deflate, gzip',
'Accept-Language' : 'it_IT'
}

url = "https://stream.twitter.com/1/statuses/filter.json"
#r = requests.get(url, headers = headers) #this triggers keep-alive: True
s = requests.Session()
r = requests.Request('GET', url, headers)
allergique
  • 73
  • 5