Why does Python selenium throw a StaleElementReferenceException when I try to access the text of an element?

Question

I am trying to access the text of an element using selenium with Python. I can access the elements themselves just fine, but when I try to get the text it doesn't work.

This is my code:

from selenium import webdriver
driver = webdriver.Chrome() # I removed the path for my post, but there is one that works in my actual code
URL = "https://www.costco.com/laptops.html"
driver.get(URL)
prices = driver.find_elements_by_class_name("price")
print([price.text for price in prices])

If I run this code I get: selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: element is not attached to the page document

However, if I were to print out the elements themselves, I have no problem. I read some previous posts about the stale element exception, but I don't understand why it applies to me in this case. Why would the DOM change when I try to access the text? Why is this happening?

Answer 1

Turn out you just need to wait:

from selenium import webdriver
import time
driver = webdriver.Chrome() # I removed the path for my post, but there is one that works in my actual code
URL = "https://www.costco.com/laptops.html"
driver.get(URL)
time.sleep(3)

prices = driver.find_elements_by_class_name("price")

print([price.text for price in prices])

Output:

['$1,999.99', '$2,299.99', '', '', '$769.99', '', '$799.99', '$1,449.99', '$1,199.99', '$1,199.99', '$1,999.99', '$1,599.99', '$1,299.99', '$2,299.99', '$1,549.99', '$1,499.99', '$599.99', '$1,699.99', '$1,079.99', '$2,999.99', '$1,649.99', '$1,499.99', '$2,399.99', '$1,499.97', '$1,199.99', '$1,649.99', '$849.99', '']

The correct way to do this is to use WebDriverWait. See

---

Old answer:

I am not entirely sure why that is happening. But I would suggest you try BeautifulSoup:

from selenium import webdriver
from bs4 import BeautifulSoup

driver = webdriver.Chrome() # I removed the path for my post, but there is one that works in my actual code
URL = "https://www.costco.com/laptops.html"
driver.get(URL)

soup = BeautifulSoup(driver.page_source)

divs = soup.find_all("div",{"class":"price"})

[div.text.replace("\t",'').replace("\n",'') for div in divs]

Output:

['$1,099.99',
 '$399.99',
 '$1,199.99',
 '$599.99',
 '$1,049.99',
 '$799.99',
 '$699.99',
 '$949.99',
 '$699.99',
 '$1,999.99',
 '$449.99',
 '$2,699.99',
 '$1,149.99',
 '$1,599.99',
 '$1,049.99',
 '$1,249.99',
 '$299.99',
 '$1,799.99',
 '$749.99',
 '$849.99',
 '$2,299.99',
 '$999.99',
 '$649.99',
 '$799.99']