Python Selenium: I am getting element is not attached to the page document error

Question

I am trying to build a scraper using selenium package for python but I am getting this error:

Message: stale element reference: element is not attached to the page document
  (Session info: headless chrome=83.0.4103.61)

I am using google colab. My code is:

import selenium
driver = webdriver.Chrome('chromedriver',chrome_options=chrome_options)
titles=[]
for link in links:
  driver.get(link)
  data = driver.find_elements_by_xpath('.//a[@class = "question-hyperlink"]')
  titles.append(data[0].text)

The error is in line data = driver.find_elements_by_xpath('.//a[@class = "question-hyperlink"]')

I was told to try exception handling but I am unable to implement it. Please help me with how can I implement it.

exception handling is `try` and `except` around what you want to catch - have a read here https://docs.python.org/3/tutorial/errors.html - if you have a go and post a code with a specific error we can help :-) — RichEdwards, Jul 31 '20 at 12:17

score 2 · Accepted Answer · answered Jul 31 '20 at 12:23

It seems synchronization issue while iterating list of links. Induce WebDriverWait() and wait for visibility_of_all_elements_located() and then iterate and store in the list.

Use try..except block to handle.

Code:

titles=[]
for link in links:
  driver.get(link)
  try:
      data=WebDriverWait(driver,10).until(EC.visibility_of_all_elements_located((By.XPATH,'//a[@class = "question-hyperlink"]')))
      for d in data:
        titles.append(d.text)
  except:
      print("element not found")
      continue

You need to import below libraries.

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

Any way to speed up? because size of link is 7000+ – Tushar Agrawal Jul 31 '20 at 17:10 — Tushar Agrawal, Jul 31 '20 at 17:10

score 0 · Answer 2 · answered Jul 31 '20 at 12:29

Before you attempt to interact with the desired elements you have to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following Locator Strategy:

import selenium
driver = webdriver.Chrome('chromedriver',chrome_options=chrome_options)
titles=[]
for link in links:
  driver.get(link)
  data = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//a[@class = 'question-hyperlink']")))
  titles.append(data[0].text)

Python Selenium: I am getting element is not attached to the page document error

2 Answers2

Linked

Related