closing files properly opened with urllib2.urlopen()

Question

I have following code in a python script

  try:
    # send the query request
    sf = urllib2.urlopen(search_query)
    search_soup = BeautifulSoup.BeautifulStoneSoup(sf.read())
    sf.close()
  except Exception, err:
    print("Couldn't get programme information.")
    print(str(err))
    return

I'm concerned because if I encounter an error on sf.read(), then sf.clsoe() is not called. I tried putting sf.close() in a finally block, but if there's an exception on urlopen() then there's no file to close and I encounter an exception in the finally block!

So then I tried

  try:
    with urllib2.urlopen(search_query) as sf:
      search_soup = BeautifulSoup.BeautifulStoneSoup(sf.read())
  except Exception, err:
    print("Couldn't get programme information.")
    print(str(err))
    return

but this raised a invalid syntax error on the with... line. How can I best handle this, I feel stupid!

As commenters have pointed out, I am using Pys60 which is python 2.5.4

The "with" statement is only available in Python 2.6, or in 2.5 if you put `from __future__ import with_statement` at the top of your file. I don't quite remember what Python version PyS60 implements but it could be 2.5? — Liquid_Fire, Oct 07 '10 at 10:47

score 18 · Answer 1 · answered Feb 13 '13 at 08:18

I would use contextlib.closing (in combination with from __future__ import with_statement for old Python versions):

from contextlib import closing

with closing(urllib2.urlopen('http://blah')) as sf:
    search_soup = BeautifulSoup.BeautifulStoneSoup(sf.read())

Or, if you want to avoid the with statement:

try:
    sf = None
    sf = urllib2.urlopen('http://blah')
    search_soup = BeautifulSoup.BeautifulStoneSoup(sf.read())
finally:
    if sf:
        sf.close()

Not quite as elegant though.

score 8 · Answer 2 · answered Oct 07 '10 at 12:22

8

finally:
    if sf: sf.close()

answered Oct 07 '10 at 12:22

dgarcia

249
5
10

unutbu · Accepted Answer · 2010-10-07T12:12:35.907

7

Why not just try closing sf, and passing if it doesn't exist?

import urllib2
try:
    search_query = 'http://blah'
    sf = urllib2.urlopen(search_query)
    search_soup = BeautifulSoup.BeautifulStoneSoup(sf.read())
except urllib2.URLError, err:
    print(err.reason)
finally:
    try:
        sf.close()
    except NameError: 
        pass

edited Oct 07 '10 at 12:12

answered Oct 07 '10 at 11:32

unutbu

842,883
184
1,785
1,677

Knew I was missing something obvious, nested trys! Your solution is very elegant, thanks! – fearoffours Oct 07 '10 at 12:13

score 1 · Answer 4 · answered Oct 07 '10 at 10:44

1

Given that you are trying to use 'with', you should be on Python 2.5, and then this applies too: http://docs.python.org/tutorial/errors.html#defining-clean-up-actions

answered Oct 07 '10 at 10:44

Habbie

2,150
15
17

Honsen · Answer 5 · 2017-10-30T03:33:45.760

If urlopen() has an exception, catch it and call the exception's close() function, like this:

try:
    req = urllib2.urlopen(url)
    req.close()
    print 'request {0} ok'.format(url)
except urllib2.HTTPError, e:
    e.close()
    print 'request {0} failed, http code: {1}'.format(url, e.code)
except urllib2.URLError, e:
    print 'request {0} error, error reason: {1}'.format(url, e.reason)

the exception is also a full response object, you can see this issue message: http://bugs.jython.org/issue1544

cclauss · Answer 6 · 2013-08-25T15:26:23.490

You could create your own generic URL opener:

from contextlib import contextmanager

@contextmanager
def urlopener(inURL):
    """Open a URL and yield the fileHandle then close the connection when leaving the 'with' clause."""
    fileHandle = urllib2.urlopen(inURL)
    try:     yield fileHandle
    finally: fileHandle.close()

Then you could then use your syntax from your original question:

with urlopener(theURL) as sf:
    search_soup = BeautifulSoup.BeautifulSoup(sf.read())

This solution gives you a clean separation of concerns. You get a clean generic urlopener syntax that handles the complexities of properly closing the resource regardless of errors that occur underneath your with clause.

score 0 · Answer 7 · answered Mar 24 '15 at 13:28

Why not just use multiple try/except blocks?

try:
    # send the query request
    sf = urllib2.urlopen(search_query)
except urllib2.URLError as url_error:
    sys.stderr.write("Error requesting url: %s\n" % (search_query,))
    raise

try:
    search_soup = BeautifulSoup.BeautifulStoneSoup(sf.read())
except Exception, err: # Maybe catch more specific Exceptions here
    sys.stderr.write("Couldn't get programme information from url: %s\n" % (search_query,))
    raise # or return as in your original code
finally:
    sf.close()

score 0 · Answer 8 · answered Oct 07 '10 at 11:20

0

Looks like the problem runs deeper than I thought - this forum thread indicates urllib2 doesn't implement with until after python 2.6, and possibly not until 3.1

answered Oct 07 '10 at 11:20

fearoffours

1,455
2
12
23

This is true. I'm on Python 2.7 (commenting in 2013) and it doesn't seem to work, so I'm rewriting something written originally for Python3 in Python2 and I'm forced to rewrite all my `with urllib.urlopen(source) as f:` statements. – erewok Sep 23 '13 at 20:04

closing files properly opened with urllib2.urlopen()

8 Answers8