Have a python function run for an alotted time

Question

I have a python script that pulls from various internal network sources. With how our systems are set up we will initiate a urllib pull from a network location and it will get hung up waiting forever for a response on certain parts of the network. I would like my script to check that if it hasnt finished the pull in lets say 5 minutes it will pass the function and attempt to pull from the next address, and record it to a bad directory repository(so we can go check out which systems get hung up, there's like over 20,000 IP addresses we are checking some with some older scripts running on them that no longer work but will still try and run when requested, and they never stop trying to run)

Im familiar with having a script pause at a certain point

import time
time.sleep(300)

What Im thinking from a psuedo code perspective (not proper python just illustrating the idea)

import time
import urllib2
url_dict = ['http://1', 'http://2', 'http://3', ...]
fail_log_path = 'C:/Temp/fail_log.txt'
for addresses in url_dict:
    clock_value = time.start()
    while clock_value <= 300:
        print str(clock_value)
        res = urllib2.retrieve(url)
    if res != []:
        pass
    else:
        fail_log = open(fail_log_path, 'a')
        fail_log.write("Failed to pull from site location: " + str(url) + "\n")
        faile_log.close

Update: a specific option for this dealing with urls timeout for urllib2.urlopen() in pre Python 2.6 versions

Found this answer which is more in line with the overall problem of my question: kill a function after a certain time in windows

Answer 1

Your code as is doesn't seem to describe what you were saying. It seems you want the if/else check inside your while loop. On top of that, you would want to loop over the ip addresses and not over a time period as your code is currently written (otherwise you will keep requesting the same ip address every time). Instead of keeping track of time yourself, I would suggest reading up on urllib.request.urlopen - specifically the timeout parameter. Once set, that function call will throw a socket.timeout exception once the time limit is reached. Surround that with a try/except block catching that error and then handle it appropriately.