6

I have made a program using urllib2 that makes a lot of connections across the web. I noticed that eventually that this can be DDoS worthy; I would like to know how to close down each connection after I have done my business to prevent such an attack.

The code I am using to open a connection is:

cj = cookielib.CookieJar()
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
r = opener.open("http://www.python.org)
html = r.read()
Jonathan Leffler
  • 730,956
  • 141
  • 904
  • 1,278
nobody
  • 71
  • 1
  • 1
  • 3
  • Look at this related question: http://stackoverflow.com/questions/1522636/should-i-call-close-after-urllib-urlopen but pay attention to 2nd and 3rd answers which are more accurate. – Senthil Kumaran Jul 06 '11 at 00:45

3 Answers3

8

I assume you are opening them with the urlopen() function. Its documentation states:

This function returns a file-like object with two additional methods:

As a file-like object, it will have a close method which you can call:

connection = urllib2.urlopen(url)
# Do cool stuff in here.
connection.close()

Update: Using the code you added to your question:

>>> import urllib2
>>> import cookielib
>>> cj = cookielib.CookieJar()
>>> opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
>>> r = opener.open("http://www.python.org")
>>> html = r.read()
>>> r.close??
Type:  instancemethod
Base Class: <type 'instancemethod'>
String Form: <bound method addinfourl.close of <addinfourl at 150857644 whose fp = <socket._fileobject object at 0x8fd48ec>>>
Namespace: Interactive
File:  /usr/lib/python2.6/urllib.py
Definition: r.close(self)
Source:
    def close(self):
        self.read = None
        self.readline = None
        self.readlines = None
        self.fileno = None
        if self.fp: self.fp.close()
        self.fp = None

So the close() method exists and actually does something:

>>> r.close()
>>> r.read()
------------------------------------------------------------
Traceback (most recent call last):
  File "<ipython console>", line 1, in <module>
TypeError: 'NoneType' object is not callable
Blair
  • 15,356
  • 7
  • 46
  • 56
  • 1
    @nobody - not on my machine. I updated my answer to use the code you added to your question. – Blair Jul 05 '11 at 23:50
2

your question is extremely vague.

but here is an example of closing a connection after use:

f = urllib2.urlopen(req)
f.read()
f.close()
Corey Goldberg
  • 59,062
  • 28
  • 129
  • 143
  • Close is not defined; there has to be some other proper way. – nobody Jul 05 '11 at 23:32
  • 2
    close() is defined on the urlopen return (file-like object) – Corey Goldberg Jul 05 '11 at 23:33
  • the close is just `def close: pass` – Senthil Kumaran Jul 05 '11 at 23:36
  • 1
    @Senthil Kumaran - not on my machine (Ubuntu 10.10/Python 2.6.6) it isn't. The comment formatting will make this a bit hard to read, but on my machine it does ``self.read = None; self.readline = None; self.readlines = None; self.fileno = None; if self.fp: self.fp.close(); self.fp = None`` i.e., it closes the underlying file object and overwrites the read methods with ``None`` so they cannot be used. – Blair Jul 05 '11 at 23:41
0

The socket connection closes automatically, once the response is obtained. So you don't explicitly close the urllopen object, it happens automatically at the socket level.

Senthil Kumaran
  • 54,681
  • 14
  • 94
  • 131
  • No it doesn't. Set up a connection: ``conn = urllib2.urlopen('http://www.google.com/')``, and then check the status of the socket: ``print conn.fp.closed`` which will print ``False`` as you'd hope. Read the data: ``devnull = conn.read()``, and now check the status of the socket: ``print conn.fp.closed`` which still prints ``False``, i.e., the socket is still open. – Blair Jul 05 '11 at 23:54
  • Admittedly it may be different on your operating system / Python version, but this just goes to show you cannot rely on this happening implicitly. Explicit is better than implicit... – Blair Jul 05 '11 at 23:55
  • 1
    The connection actually is closed whenever you `read()` the response, you can verify this by looking at the `read()` method of `HTTPResponse`, using `netstat`, etc. – Zach Kelling Jul 06 '11 at 00:42