How to get text from div class using Selenium and find_element() method

Question

I'm trying to get the values ​​inside the div, after finishing a scroll in selenium. The scroll part works correctly, but I can't print the values ​​of the div class.

My code:

import time
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.service import Service


s=Service('D:\enviroments\casino\chromedriver.exe')
driver = webdriver.Chrome(service=s)
url='https://www.casinoimportaciones.com.uy/accesorios-escola'
driver.get(url)

time.sleep(5)
# Get scroll height
last_height = driver.execute_script("return document.body.scrollHeight")

while True:
    # Scroll down to bottom
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

    # Wait to load page
    time.sleep(5)

    # Calculate new scroll height and compare with last scroll height
    new_height = driver.execute_script("return document.body.scrollHeight")
    if new_height == last_height:
        driver.execute_script("window.scrollTo(0,-250);")
        time.sleep(10)
        elem2=driver.find_element(By.CLASS_NAME,"//div[@class='description']")
        
        print(elem2)
        time.sleep(100)
    else:
        last_height = new_height

Output error:

---

Update

Update: After fix find_element, he print a list of elements, but no the text:

Answer 1

The error you're receiving may be due to you're searching by class name but you're using an XPATH. Personally I like to use By.CSS_SELECTOR but to each their own. Pick one and I believe you can get the value from the element using one of these two ways:

Example given with XPATH since you're using an XPATH.

elem2=driver.find_element(By.XPATH,"//div[@class='description']").text

Or you can try:

elem2=driver.find_element(By.XPATH,"//div[@class='description']")
elem2.get_attribute("value")

You may also need to wait for the element to actually be in the DOM to get the text/value. You can use WebDriverWait to wait for the element to either be in the DOM or be in the DOM and visible.

Answer 2

This error message...

...implies that InvalidSelectorException was raised as an invalid / illegal selector was supplied.

---

This usecase

In the line:

elem2=driver.find_element(By.CLASS_NAME,"//div[@class='description']")

though you mentioned about By.CLASS_NAME but in the value field you have passed a xpath value instead of a classname.

---

Solution

You need to change the line of code in following either of the following locator strategies:

• Using By.CLASS_NAME:

  elem2 = driver.find_element(By.CLASS_NAME, "description")
  

• Using By.XPATH:

  elem2 = driver.find_element(By.XPATH, "//div[@class='description']")
  

• Using By.CSS_SELECTOR:

  elem2 = driver.find_element(By.CSS_SELECTOR, "div.description")
  

---

Update

As per your question update these

...are the elements printed to the console as per the line of code:

print(elem2)

Looking at the HTML DOM for the website possibly you are looking for the product description and in that case you can use either of the following solutions:

• Using text attribute:

  print(elem2.text)
  

• Using get_attribute("textContent"):

  print(elem2.get_attribute("textContent"))
  

• Using get_attribute("innerHTML"):

  print(elem2.get_attribute("innerHTML"))