0

I am currently learning to scrape data from websites with python. The kind of websites I am working with are slow and stop working multiple times during the day due to which I am having to use try/catch often.

For e.g.: (driver in this case if the chrome webdriver with selenium)

time.sleep(2)
try:
    dist_lst = [i.text for i in driver.find_element_by_xpath('//*[@id="cphBody_GridArrivalData"]').find_elements_by_tag_name('tr')]
except:
    input("Check if data is loaded")
    dist_lst = [i.text for i in driver.find_element_by_xpath('//*[@id="cphBody_GridArrivalData"]').find_elements_by_tag_name('tr')]

I'm wondering which would be the "right" way to do it and why:

  1. Use if statement to check if: data is loaded then do the list comprehension else: ask user to check
  2. Use try: to do the list comprehension anyway if it fails then ask user to check if data is loaded and then do list comprehension again
  3. Is there a "better" way to do this?

I'm new to python and coding in general so I also wanted to know how would you determine the "right way"? Are there some kind of programming norms or like some say pythonic way to determine how to go about the process?

P.S: I've read that list comprehension is slow in python so if I wasn't using list comprehension in the example code would your answer change?

Canute S
  • 334
  • 1
  • 5
  • 12
  • Side note: you probably don't want to be [using a bare except clause](https://stackoverflow.com/questions/54948548/what-is-wrong-with-using-a-bare-except). This will catch _all_ errors, including those that aren't derived from `Exception`, like `KeyboardInterrupt` or `SyntaxError`. – Brian61354270 Jun 18 '20 at 17:28

1 Answers1

0

Use a while loop, breaking if you don't catch an exception.

while True:
    try:
         dist_lst = [...]
         break
    except Exception:
        input("Check if data is loaded")
chepner
  • 497,756
  • 71
  • 530
  • 681