Web scraping loop problem - Element not attached to the page document

Question

I want to download the game logs CSV file for all the skaters for the seasons 2015 to 2019 of: https://evolving-hockey.com/

However, there is an error message that pops up at different times in the for loop. StaleElementReferenceException: Message: stale element reference: element is not attached to the page document

I look on the subject and what I found was that it is because when the webpage is refreshed during the loop the element is no longer in the DOOM or changed... But I did not find anything to correct it in my case. I try to add some time.sleep, but I still get the error. Here is my code:

from selenium import webdriver
import csv
from selenium.webdriver.support.ui import Select
from datetime import date, timedelta
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait 
from selenium.webdriver.support import expected_conditions as EC
import time
from selenium.webdriver.common.keys import Keys
from selenium.common.exceptions import TimeoutException

chromedriver =("C:/Users/Michel/Desktop/python/package/chromedriver_win32/chromedriver.exe")
driver = webdriver.Chrome(chromedriver)
driver.get("https://evolving-hockey.com/")

#Click Games and then game logs
Gamestab= driver.find_element_by_xpath("/html/body/nav/div/ul/li[6]/a")
Gamestab.click()
Gameslog= driver.find_element_by_xpath("/html/body/nav/div/ul/li[6]/ul/li[3]/a")
Gameslog.click()


Strenght= driver.find_element_by_xpath("//*[@id='tab-7262-1']/div/div[1]/div[3]/div/div/button")
Strenght.click()

All=driver.find_element_by_xpath("//*[@id='tab-7262-1']/div/div[1]/div[3]/div/div/div/ul/li[1]/a")
All.click()


Totals=driver.find_element_by_xpath("//*[@id='game_logs_skaters_stat_display']/div[2]/div[1]")
Totals.click()



# Loop all teams and all seasons
# ## TEAM

for b in range(1,2340):
    time.sleep(5)
    Player= driver.find_element_by_xpath("//*[@id='tab-7262-1']/div/div[1]/div[1]/div/div/div/div[1]")
    time.sleep(5)
    Player.click()
    Playername= driver.find_element_by_xpath("//*[@id='tab-7262-1']/div/div[1]/div[1]/div/div/div/div[2]/div/div[%d]" %(b))
    time.sleep(5)
    Playername.click()


# # ## Season- 20152016to20182019

    for i in range(1,5):
        Season=driver.find_element_by_xpath("//*[@id='tab-7262-1']/div/div[1]/div[2]/div/div/button")
        time.sleep(5)
        Season.click()
        time.sleep(5)
        Season1819=driver.find_element_by_xpath("//*[@id='tab-7262-1']/div/div[1]/div[2]/div/div/div/ul/li[%s]" %(i))
        time.sleep(5)
        Season1819.click()

## SUBMIT
        submit = driver.find_element_by_id('game_logs_skaters_submit_button')
        submit.click()
        time.sleep(10)

# # Click download

        download = driver.find_element_by_id('game_logs_skaters_download')
        download.click()


driver.close()

Once your element is stale (most likely because of a refresh), you cannot re-use it, you need to call find_element_by* again to retrieve it as the preview element id is not attached to the DOM anymore. — Nic Laforge, Mar 11 '19 at 23:56

score 1 · Accepted Answer · answered Mar 12 '19 at 00:13

StaleElement exception occur when the element is either not part of the DOM anymore or got refresh.

One solution for your case would be to implement a method that would retry

from selenium.common.exceptions import StaleElementReferenceException

def click_element(driver, locator, value):
   try:
       driver.find_element(locator, value).click()
   exception StaleElementReferenceException:
       driver.find_element(locator, value).click()

In your code you can now call the click_element method above:

click_element(driver, "xpath", "//*[@id='tab-7262-1']/div/div[1]/div[1]/div/div/div/div[1]")

This should work for your case as from your code you are only clicking. But if you need to perform more actions (i.e send_keys, get_attribute(), text) you might think about implementing a wrapper around every selenium calls and implement this kind of try/catch retry mechanism.

Should I use the click_element everywhere in my code that I am clicking ? How I should use the Xpath that I am looping through with [%d] %(b) ? — Jagr, Mar 12 '19 at 15:13
If some of your element are static, meaning that they don't get internally refreshed you can use click() directly. It won't harm if you always use the above method either. For your xpath, you can pass the same value directly to the method. ```(i.e: value="//*[@id='tab-7262-1']/div/div[1]/div[1]/div/div/div/div[2]/div/div[%d]" %(b))``` — Nic Laforge, Mar 12 '19 at 15:35

Web scraping loop problem - Element not attached to the page document

1 Answers1