Skip URL if timeout

Question

I have a list of URL's

I am using the following to retrieve their contents:

for url in url_list:
    req = urllib2.Request(url)
    resp = urllib2.urlopen(req, timeout=5)
    resp_page = resp.read()
    print resp_page

When there is a timeout, the program just crashes. I just want to read the next URL if there is a socket.timeout: timed out. How to do this?

Thanks

For a similar problem see: http://stackoverflow.com/questions/2712524/handling-urllib2s-timeout-python — Raz, Nov 09 '11 at 22:31

score 7 · Answer 1 · edited Nov 24 '12 at 10:11

7

Although there already is an answer, I'd like to point out that URLlib2 might not be the sole responsible with this behavior.

As pointed out here (and as it also seems based on the problem description), the exception may belong to the socket library.

In that case just add another except:

import socket

try:
    resp = urllib2.urlopen(req, timeout=5)
except urllib2.URLError:
    print "Bad URL or timeout"
except socket.timeout:
    print "socket timeout"

edited Nov 24 '12 at 10:11

RajeshM

872
11
21

answered Jan 04 '12 at 14:44

Jir

2,985
8
44
66

2

Downvoting with no explanation is not really useful, is it? – Jir Nov 20 '12 at 11:15

score 1 · Accepted Answer · answered Nov 09 '11 at 22:41

I'm going to go ahead and assume that by "crashes" you mean "raises a URLError", as described by the urllib2.urlopen docs. See the Errors and Exceptions section of the Python Tutorial.

for url in url_list:
    req = urllib2.Request(url)
    try:
        resp = urllib2.urlopen(req, timeout=5)
    except urllib2.URLError:
        print "Bad URL or timeout"
        continue # skips to the next iteration of the loop
    resp_page = resp.read()
    print resp_page

score 1 · Answer 3 · answered Nov 09 '11 at 22:42

Sounds like you just need to catch the timeout exception. I don't get a socket.timeout message that you do.

req = urllib2.Request("http://127.0.0.2")
try:
    resp = urllib2.urlopen(req, timeout=5)
except urllib2.URLError:
    print "Timeout!"

Obviously, you need to have a URL that will actually timeout (127.0.0.2 may not on your box).

Skip URL if timeout

3 Answers3