Persistent HTTPS Connections in Python

Question

I want to make an HTTPS request to a real-time stream and keep the connection open so that I can keep reading content from it and processing it.

I want to write the script in python. I am unsure how to keep the connection open in my script. I have tested the endpoint with curl which keeps the connection open successfully. But how do I do it in Python. Currently, I have the following code:

c = httplib.HTTPSConnection('userstream.twitter.com')
c.request("GET", "/2/user.json?" + req.to_postdata())
response = c.getresponse()

Where do I go from here?

Thanks!

score 3 · Answer 1 · answered Aug 26 '11 at 06:23

It looks like your real-time stream is delivered as one endless HTTP GET response, yes? If so, you could just use python's built-in urllib2.urlopen(). It returns a file-like object, from which you can read as much as you want until the server hangs up on you.

f=urllib2.urlopen('https://encrypted.google.com/')
while True:
    data = f.read(100)
    print(data)

Keep in mind that although urllib2 speaks https, it doesn't validate server certificates, so you might want to try and add-on package like pycurl or urlgrabber for better security. (I'm not sure if urlgrabber supports https.)

score 1 · Answer 2 · answered Aug 26 '11 at 01:34

1

Connection keep-alive features are not available in any of the python standard libraries for https. The most mature option is probably urllib3

answered Aug 26 '11 at 01:34

SingleNegationElimination

151,563
33
264
304

this is false. urllib2 from stdlib doesn't support persistent connections but [httplib does support it](http://stackoverflow.com/a/9781027/4279). Though urllib3 or [requests (built on top of urllib3)](http://docs.python-requests.org/en/latest/index.html) are not bad options if you can use 3rd party libraries. – jfs Aug 26 '12 at 19:46

score 0 · Answer 3 · answered Aug 26 '11 at 09:02

httplib2 supports this. (I'd have thought this the most mature option, didn't know urllib3 yet, so TokenMacGuy may still be right)

EDIT: while httplib2 does support persistent connections, I don't think you can really consume streams with it (ie. one long response vs. multiple requests over the same connection), which I now realise you may need.

Persistent HTTPS Connections in Python

3 Answers3

Linked