0

I'm trying to scrape some urls, but depending on the amount of results, the page code changes. Both below are working, but none returns me 100% of the results. I'm trying to create a 'if' of some sort that will leave no NAN results.

url = [url.get_attribute("href") for url in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CLASS_NAME, 'base-card__full-link')))]

and

url = [url.get_attribute("href") for url in WebDriverWait(driver, 50).until(EC.visibility_of_all_elements_located((By.XPATH, '//*[@id="main-content"]/section[2]/ul/li/a')))]

Would much appreciate your help.

undetected Selenium
  • 183,867
  • 41
  • 278
  • 352
Owl J
  • 1
  • 1
  • This sounds like an [X-Y problem](http://xyproblem.info/). Instead of asking for help with your solution to the problem, edit your question and ask about the actual problem. What are you trying to do? – undetected Selenium Dec 21 '21 at 13:40

1 Answers1

0

To create a list of the href attributes you can clubup the conditions using within a lambda expression and you can use either of the following Locator Strategies:

  • Using lambda expression:

    hrefs = [url.get_attribute("href") for url in WebDriverWait(driver, 20).until(lambda driver: driver.find_element(By.CLASS_NAME,"base-card__full-link") or driver.find_element(By.XPATH,"//*[@id='main-content']/section[2]/ul/li/a"))]
undetected Selenium
  • 183,867
  • 41
  • 278
  • 352
  • Thank you. I totally understood what you did there, but it's not working. It's returning "'WebElement' object is not iterable" so it's not skipping the not found itens somehow. – Owl J Jan 11 '22 at 14:02