Python Better network API than urllib

Question

I noticed there is no way to close connections in urllib (close does not work). Is there a better more robust network API for python?

Answer 1

You are not alone with the problems of urllib. Python community has come up with few alternatives

Try here:

http://pypi.python.org/pypi/requests

Requests is sane API over urllib

http://urlgrabber.baseurl.org/

Urlgrabber is highspeed rewrite of urllib supporting advanced HTTP download functionalities.

Answer 2

Urllib closes connection itself after finishing urlopen, fp.close() just closes filebuffer, which holds retrieved info:

>>> import urllib
>>> fp = urllib.urlopen('http://www.httpbin.org/ip')
>>> fp.read()
0: '{"origin": "::ffff:92.242.181.219"}'
>>> fp.close()

There are many good http libraries:

• requests - easy http-client, built around urlib2/httplib
• tornado asyncclient - very light and async interface, mainly to make simple REST requests.

• pycurl - fastest and most powerful networking library, supports another protocols, there also exist curls modules,which are ported to tornado and gevent

• i am currently building profilers for those libraries: source - there will be speed and memory comparison also.

another choices:

• twisted webclient - grownup async library
• urllib2 - traditional library for opening URLs
• httplib - HTTP protocol client
• Doug Hellman's list of Internet and networking modules

Answer 3

To close the connection you can use the with statement. In that way you ensure a proper closure of the connection, closing even in a case of exception. See this for a better explanation

Answer 4

http://docs.python.org/library/urllib2.html

http://docs.python-requests.org/en/latest/index.html

Answer 5

Since it isn't mentioned yet: httplib2 is great (although one of it's great points is reusing persistent (keep-alive) connections rather than closing them)

As for closing the connection: what is it exactly that you're trying to achieve?

If you want to have the connection closed after the request (or rather, after you get the response), you could add a Connection: close header to your request. (see http spec). This will cause the server to close the connection (if it's a well-behaved server at least. I think with httplib2 this will also cause the connection to be closed (by the client) in case server doesn't behave as expected. I don't know about the other libs)