StaleElementException on Selenium Loop

Question

I'm scraping a directory website for urls and I keep getting a StaleElementException whenever the next button is no longer valid. I'm trying to figure out how to prevent the loop from breaking. I'd like to exit the loop after getting the rest of the elements and when there is no longer a next button.

with open(provider + '-' + state + '-' + city + '-links.csv', 'w', newline='') as file:
    writer = csv.writer(file)
    idx = 1
    while next_page is not None:
        for company in company_links_elements:
            company_url = company.get_attribute("href")
            writer.writerow((idx, company_url, provider))
            idx += 1
        time.sleep(random.randint(2, 3))
        next_page.click()
        # Get next page elements
        company_links_elements = driver.find_elements(By.XPATH,
                                                      "//h3[@class='jss320 jss324 jss337 sc-gzOgki eucExu']/a")
        company_address_elements = driver.find_elements(By.XPATH,
                                                        "//p/strong[@class='dtm-search-listing-address']")
        # Try getting the next page element
        try:
            next_page = driver.find_element(By.XPATH, "//a[@role='link'][contains(text(),'Next')]")
        except NoSuchElementException:
            break

driver.quit()

This is the error:

raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: element is not attached to the page document
  (Session info: chrome=66.0.3359.181)
  (Driver info: chromedriver=2.38.552522 (437e6fbedfa8762dec75e2c5b3ddb86763dc9dcb),platform=Windows NT 10.0.16299 x86_64)

I tried putting a try brake whenever I load a new page for it to fail over if it doesn't exist, but this doesn't seem to be working.

Two possible regions of this exception are - i)The element has been deleted entirely. ii) The element is no longer attached to the DOM. — Kuldeep Yadav, May 21 '18 at 05:02
Check this discussion https://stackoverflow.com/questions/44838538/staleelementreference-exception-in-pagefactory/44844879#44844879 — undetected Selenium, May 21 '18 at 05:45
If looping through a set of `WebElements` requires you to navigate to a new page within the loop, the loop will not work. Since you are locating using XPath, you could loop through that with XPath indexes instead — Mangohero1, May 21 '18 at 11:41

score 0 · Answer 1 · answered May 21 '18 at 02:46

0

Try add a long sleep/wait after next_page.click() for debug purpose to wait the page refresh complete after click next page button

answered May 21 '18 at 02:46

yong

13,357
1
16
27

StaleElementException on Selenium Loop

1 Answers1