4

For now I am doing this: (Python3, urllib)

url = 'someurl'
headers = '(('HOST', 'somehost'), /  
            ('Connection', 'keep-alive'),/
            ('Accept-Encoding' , 'gzip,deflate'))
opener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor())
for h in headers:
    opener.addheaders.append(x)
data = 'some logging data' #username, pw etc.
opener.open('somesite/login.php, data)

res = opener.open(someurl)
data = res.read()
... some stuff here...
res1 = opener.open(someurl2)
data = res1.read()
etc.

What is happening is this;

I keep getting gzipped responses from server and I stayed logged in (I am fetching some content which is not available if I were not logged in) but I think the connection is dropping between every request opener.open;

I think that because connecting is very slow and it seems like there is new connection every time. Two questions:

a)How do I test if in fact the connection is staying-alive/dying
b)How to make it stay-alive between request for other urls ?

Take care :)

pyfunc
  • 65,343
  • 15
  • 148
  • 136
Piotr Lopusiewicz
  • 2,514
  • 2
  • 27
  • 38
  • a) Sniffing the traffic would be a trivial way to check whether you're keeping-alive. Make sure the server supports keepalive as well. As for b), at least in python2.x there was a way to specify a handler that could keepalive, but it didn't seem default. – Robert Dec 08 '10 at 08:12
  • 1
    duplicate http://stackoverflow.com/questions/1037406/python-urllib2-with-keep-alive – khachik Dec 08 '10 at 08:12
  • Well, first this is python3 and this urlgrabber recommended in the other thread is supposedly for 2.5 and before and second I would prefer to avoid using external libraries for this seemingly simple task; especially because i want to understand how those things work – Piotr Lopusiewicz Dec 08 '10 at 08:53

2 Answers2

1

This will be a very delayed answer, but:

You should see urllib3. It is for Python 2.x but you'll get the idea when you see their README document.

And yes, urllib by default doesn't keep connections alive, I'm now implementing urllib3 for Python 3 to be staying in my toolbag :)

keremulutas
  • 534
  • 5
  • 16
0

Just if you didn't know yet, python-requests offer keep-alive feature, thanks to urllib3.

Tankman六四
  • 1,638
  • 2
  • 13
  • 19