How to use time module on a for loop to prevent delays in list processing

Question

How can I use the time module to skip to the next iteration if it's taking longer than 5 seconds? Specifically what is the correct way to implement the solution from How would I stop a while loop after n amount of time? with the for loop I'm using below?

import requests
from bs4 import BeautifulSoup

for n in random_list:
    url = all_raw_urls[n]
    try:
        req = requests.get(url)
        data = req.text
        soup = BeautifulSoup(data, 'html.parser')

        tags = soup.find_all('img')
        tags = list(set(tags))

        if len(tags) < 15 or len(tags) > 50:
            print(str(image_count) + ': leave' + ' : images: ' + str(len(tags)))
        else:
            print(str(image_count) + ': keep' ' : images: ' + str(len(tags)))
            image_urls.append(url)
    except:
        print('request error')

    image_count += 1

score 1 · Accepted Answer · answered Nov 22 '16 at 07:44

You can use requests embedded timeouts:

req = requests.get(url, timeout=5)

Value of timeout parameter is the number of seconds to wait for request or a tuple (connect_timeout, read_timeout):

req = requests.get(url, timeout=(0.5, 5))

If you really want to use time module, then you can put everything from try/except into another thread and in while loop check its state after 5 seconds, if thread is not finished, kill it and run next task in another thread.

How to use time module on a for loop to prevent delays in list processing

1 Answers1