1
import sys
import urllib2
import time
from bs4 import BeautifulSoup
from selenium import webdriver
import string
import re

reload(sys)
sys.setdefaultencoding('utf8')

baseUrl = 'https://www.breastsurgeons.org/new_layout/membership/membersearch/index.php'

driver = webdriver.Chrome('/usr/local/Cellar/chromedriver/2.36/bin/chromedriver')
driver.get(baseUrl)
time.sleep(20)

for p in range(1,282):

    driver.find_element_by_xpath("//a[contains(text(),'>>')]").click()
    time.sleep(2)

driver.quit()

After opening the baseUrl, I manually click agree and then search for a list of physicians to show up. I want to flip through the listing. Right now Selenium only clicks the first time by finding ">>". After which it stops and gives following error.

   driver.find_element_by_xpath("//a[contains(text(),'>>')]").click()


  File "/Library/Python/2.7/site-packages/selenium-3.11.0-py2.7.egg/selenium/webdriver/remote/webelement.py", line 80, in click


    self._execute(Command.CLICK_ELEMENT)


  File "/Library/Python/2.7/site-packages/selenium-3.11.0-py2.7.egg/selenium/webdriver/remote/webelement.py", line 628, in _execute


    return self._parent.execute(command, params)


  File "/Library/Python/2.7/site-packages/selenium-3.11.0-py2.7.egg/selenium/webdriver/remote/webdriver.py", line 312, in execute


    self.error_handler.check_response(response)


  File "/Library/Python/2.7/site-packages/selenium-3.11.0-py2.7.egg/selenium/webdriver/remote/errorhandler.py", line 242, in check_response


    raise exception_class(message, screen, stacktrace)


selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: element is not attached to the page document
undetected Selenium
  • 183,867
  • 41
  • 278
  • 352
leonard teng
  • 13
  • 2

1 Answers1

0

The error says it all :

selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: element is not attached to the page document

In your program within the for() loop you are locating the <a> tag element with text as >> on a page and invoking click() and due to the click() event the HTML DOM changes. When your program iterates the for() loop for the second time perhaps the WebElement identified as driver.find_element_by_xpath("//a[contains(text(),'>>')]") doesn't gets loaded but Selenium tries to refer the search from the previous iteration which have already turned stale. Hence you see StaleElementReferenceException.

Solution

A convincing way to iterate the pages would be instead of :

driver.find_element_by_xpath("//a[contains(text(),'>>')]").click()

You can induce WebDriverWait in-conjunction with expected_conditions clause set to element_to_be_clickable for the WebElement with the particular Page Number (e.g. 3, 4, 5, etc) to be clickable as follows :

WebDriverWait(self.driver, 30).until(EC.element_to_be_clickable((By.XPATH, "//a[contains(text(),'3')]"))).click()
undetected Selenium
  • 183,867
  • 41
  • 278
  • 352