find_elements not selecting all the elements (python)

Question

I'm trying to get all the elements with bg-primary class and save their href (hyperlink) value in a variable. Here is my code:

elems = driver.find_elements(By.CLASS_NAME , "bg-primary")
    
    for elem in elems:
        if (elem.get_attribute("href")):
            print(elem.get_attribute("href"))

And this is the website's code which has class: "bg-primary" I'm trying to search for it yet I'm not getting any results.

That's what I tried. This code block is dynamically activated on the site and I have WebDriverWait elements for waiting on the element to appear.

Answer 1

If the elements with the "bg-primary" class are dynamically activated on the site, you might face a timing issue where the elements are unavailable when you try to find them using Selenium. To resolve this, you can use WebDriverWait to wait for the elements to become visible before retrieving their href values. Here's how you can modify your code:

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

# Wait for the elements with "bg-primary" class to become visible
elems = WebDriverWait(driver, 10).until(EC.visibility_of_all_elements_located((By.CLASS_NAME, "bg-primary")))

for elem in elems:
    if elem.get_attribute("href"):
        print(elem.get_attribute("href"))

By using WebDriverWait with EC.visibility_of_all_elements_located, the code will wait for up to 10 seconds for the elements with the "bg-primary" class to become visible on the page before attempting to retrieve their href values. This should ensure that you get all the elements you are looking for. If the elements are still not selected, you might need to adjust the waiting time or use different expected conditions based on the behavior of the website.

Answer 2

Given the HTML:

you have to make a few adjustments as follows:

• The class attribute of the <a> element contains multiple values and bg-primary is one of them.
• Additionally the elements looks to be dynamic element.

---

Solution

To extract all the elements with bg-primary class and save their href value you have to induce WebDriverWait for visibility_of_all_elements_located() and you can use either of the following locator strategies:

• Using CSS_SELECTOR:

  print([my_elem.get_attribute("href") for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "a.bg-primary")))])
  

• Using XPATH:

  print([my_elem.get_attribute("href") for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//a[contains(@class, 'bg-primary')]")))])
  

• Note : You have to add the following imports :

  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support import expected_conditions as EC