0

I have a python scraping script.

At many points, the code has to interact with websites.

Sometimes the code crashes because my wifi signal drops.

I have been told that I should use: 

try: except urllib2.URLError as e:

as a block around the problem parts of the code. But the code has many lines that interact with websites, with other things in between.

My question is, do I need to use this block around every line of code that interacts with a website? Or is there something else fancy that I could do that would just tell the whole script "if you're having a problem connecting to the internet at any point, just wait 10 seconds and try that line again".

user1551817
  • 6,693
  • 22
  • 72
  • 109
  • 1
    Is there a main loop or other outer scope? One bigger error handler on the outside may make more sense than a bunch of small inner ones. This is *very* dependent on the structure of your individual program, however; there isn't really an easy generic answer. – Charles Duffy Feb 09 '19 at 01:53
  • The lines that interact are all within a big loop with other things going on. I don't want to restart the whole loop. So it seems like I would have to use the try/except on each individual line? – user1551817 Feb 09 '19 at 01:56
  • 1
    When you say you don't want to "restart the whole loop", do you mean you don't want to restart it from the first item it iterates over, or don't want to restart it even from the top for the current item? – Charles Duffy Feb 09 '19 at 02:00
  • 2
    And of course you can wrap all your functions that interact with external sites to go through a single function that has your fallback/retry logic. See f/e https://www.peterbe.com/plog/best-practice-with-retries-with-requests, and/or the existing SO question https://stackoverflow.com/q/23267409/14122 – Charles Duffy Feb 09 '19 at 02:01
  • The loop actually only runs once - I just don't want to start the loop again (which would be essentially like starting the whole code again). I will check out the links you provided to learn how I can better write the code. Thanks! – user1551817 Feb 09 '19 at 02:14
  • My point is that with appropriate structuring, you could restart the *current iteration* of the loop only. – Charles Duffy Feb 09 '19 at 13:19

0 Answers0