Python Selenium: How to click on see-more buttons in a loop

Question

I am trying to scrap the website https://www.wlw.de/de/suche?employeeCounts=200%2B_50-199&q=blechbearbeitung&supplierTypes=Dienstleister_Hersteller with multiple repetitive divs elements that each has a see-more button to show the hidden data in the div I tried to find buttons using Selenium in Python with this code

for i in range(len(company_names)):
    seemore_button = driver.find_elements(By.CLASS_NAME, 'toggle-button')[i]
    seemore_button.click()
    time.sleep(2)

Where company_names is the name of each div and I am using its length to loop on all divs but it doesn't work, when I run this code the chrome driver acts as clicking on buttons but when it finishes I find that not all the buttons have been clicked I thought at first that it may be because of the page can't load as fast as the code clicks the buttons so I added a sleep time of 2 seconds and the same result, sometimes the buttons are clicked and sometimes no

Answer 1

To click on all the show more links in a loop you can use either of the following locator strategies:

• Using CSS_SELECTOR:

  driver.get('https://www.wlw.de/de/suche?employeeCounts=200%2B_50-199&q=blechbearbeitung&supplierTypes=Dienstleister_Hersteller')
  WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a#CybotCookiebotDialogFooterButtonAcceptAll>button"))).click()
  elements = WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "a.toggle-button span")))
  for elem in elements:
      elem.click()
  

• Using XPATH:

  driver.get('https://www.wlw.de/de/suche?employeeCounts=200%2B_50-199&q=blechbearbeitung&supplierTypes=Dienstleister_Hersteller')
  WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//a[@id='CybotCookiebotDialogFooterButtonAcceptAll']/button"))).click()
  elements = WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//a[@class='toggle-button']//span")))
  for elem in elements:
      elem.click()
  

• Note : You have to add the following imports :

  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support import expected_conditions as EC