0

I'm trying to use Selenium to scrape google maps unfortunatly its's not quite working,the element is not present on page load and is added after clicking on some button but it seems that the element is not always loaded when looking for it. (I'm talking about the carousel's items that appears after clicking on a shop,restaurant while doing a specific search)

I already tried to use the classic Selenium wait options.

Things I tried:

  • time.sleep()
  • WebDriverWait(driver,30).until(EC.visibility_of_element_located(...)
  • WebDriverWait(driver,30).until(EC.element_to_be_clickable(...)
  • WebDriverWait(driver,60).until(EC.presence_of_all_elements_located(...)

Even with theses things results are random, I sometimes can access the element via Selenium and sometimes I don't. It seem's that even with long waits, Selenium can't always access the web element even tho I can click it and see it.

DisappointedByUnaccountableMod
  • 6,656
  • 4
  • 18
  • 22
seb_philippe
  • 1
  • 1
  • 1

1 Answers1

0

You can make the web driver wait until the element is clickable. Have you tried that?

element = WebDriverWait(driver, 20).until(
EC.presence_of_element_located((By.ID, "myElement")))

or a better way of doing this as pointed out here

element = WebDriverWait(driver, 20).until(
EC.element_to_be_clickable((By.XPATH, "myXpath")))
  • Yes I tried that and it seem's that it's less working that a simple time.sleep(), I always get a timeout exception even when I use long delays. – seb_philippe Sep 21 '21 at 08:31
  • Are you sure that your element is present on the page? and the XPATH is correct. You can also use element by id in selenium if you cannot figure out the proper XPATH –  Sep 21 '21 at 12:41
  • Yes the Xpath is good because it works sometimes but not every time, so I just rescrap until the webelement is found,it's not very time efficient but it works ¯\_(ツ)_/¯. And I switch between Class selector,Xpath etc ... but it's still random. The element have no Id or Name, and class might change often so I'm kinda stuck with ClassName or Xpath – seb_philippe Sep 21 '21 at 16:33
  • Try executing a javascript script using selenium to check if the DOM is ready. Or run a loop to check if the element is present else put the code to sleep. –  Sep 22 '21 at 11:49