How to get contents for elements that have the same class

Question

I'm trying to extract product informatin using selenium. Here is the URL for the page https://www.dell.com/en-us/shop/dell-laptops/sr/laptops/11th-gen-intel-core?appliedRefinements=23775

To start off, I got the parent class of the elements that I'm trying to scrape, which are the computer models, CPU, etc, and they are inclosed in cards

The parent class for the cards "stack-system ps-stack", but when I try to find the list of elements in the class, it's empty.

driver = webdriver.Chrome()
url = "https://www.dell.com/en-us/shop/dell-laptops/sr/laptops/11th-gen-intel-core?appliedRefinements=23775"
classname_main = "stack-system ps-stack"
driver.get(url)
driver.implicitly_wait(50)
products = driver.find_elements_by_class_name("stack-system ps-stack")
print(products)

I would like to get the contents for the cards as well.

Answer 1

The locator class_name won't accept spaces or multi-classes for instance. Change to use css_selector:

driver.find_elements_by_css_selector(".stack-system.ps-stack")

Answer 2

To extract the product names e.g. New Inspiron 14 5000 Laptop, etc using Selenium and python you can use either of the following Locator Strategies:

• Using css_selector and get_attribute("innerHTML"):

  print([my_elem.get_attribute("innerHTML") for my_elem in driver.find_elements_by_css_selector("li.Fruit")])
  

• Using xpath and text attribute:

  print([my_elem.text for my_elem in driver.find_elements_by_xpath("//li[@class='Fruit']")])
  

---

Ideally you need to induce WebDriverWait for visibility_of_all_elements_located() and you can use either of the following Locator Strategies:

• Using CSS_SELECTOR and get_attribute("innerHTML"):

  print([my_elem.get_attribute("innerHTML") for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "article.stack-system.ps-stack h3 > a")))])
  

• Using XPATH and text attribute:

  print([my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//article[@class='stack-system ps-stack']//h3/a")))])
  

• Console Output:

  ['New Inspiron 14 5000 Laptop', 'New Inspiron 14 5000 2-in-1 Laptop (Dune)', 'New Inspiron 15 5000 Laptop', 'New Inspiron 14 5000 Laptop', 'New Inspiron 14 5000 Laptop', 'New Inspiron 15 5000 Laptop', 'New Inspiron 14 5000 2-in-1 Laptop (Dune)', 'New Inspiron 14 5000 2-in-1 Laptop (Titan Grey)', 'New Inspiron 14 5000 2-in-1 Laptop (Titan Grey)', 'New Inspiron 13 7000 2-in-1 Laptop', 'New Inspiron 15 5000 Laptop', 'New Inspiron 15 7000 2-in-1 Laptop']
  

• Note : You have to add the following imports :

  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support import expected_conditions as EC
  

---

Outro

Link to useful documentation:

• get_attribute() method Gets the given attribute or property of the element.
• text attribute returns The text of the element.
• Difference between text and innerHTML using Selenium

Answer 3

Use this css selector to find the all text .

driver.get(url)
    
print(WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.CSS_SELECTOR,"div.no-div-lines-layout"))).text)