Selenium Element.get_attribute occasionally failing

Question

Given this code:

options = webdriver.ChromeOptions()
options.add_argument("headless")
driver = webdriver.Chrome(options=options)

driver.get('https://covid19.apple.com/mobility')

elements = driver.find_elements_by_css_selector("div.download-button-container a")

csvLink = [el.get_attribute("href") for el in elements]
driver.quit()

At the end, csvLink sometimes has the link and most times not. If I stop at the last line in the debugger, it often fails to have anything in csvlink, but if I manually execute (in the debugger) elements[0].get_attribute('href') the correct link is returned. Every time.

if I replace

csvLink = [el.get_attribute("href") for el in elements]

with a direct call -

csvLink = elements[0].get_attribute("href")

it also fails. But, again, if I'm stopped at the driver.quit() line, and manually execute, the correct link is returned.

is there a time or path dependency I'm unaware of in using Selenium?

Adding a delay solves the problem - But why?? – Marc Nov 10 '20 at 19:54 — Marc, Nov 10 '20 at 19:54

jmunsch · Accepted Answer · 2020-11-10T20:13:24.780

I'm guessing it has to do with how and when the javascript loads the link. Selenium grabs it without waiting before the javascript is able to load the elements href attribute value. Try explicitly waiting for the selector, something like:

(
    WebDriverWait(browser, 20)
    .until(EC.presence_of_element_located(
        (By.CSS_SELECTOR, "div.download-button-container a[href]")))
    .click()
)

Reference:

Also, if you curl https://covid19.apple.com/mobility my suspicion would be that the element exists ( maybe ), but the href is blank?

Selenium Element.get_attribute occasionally failing

1 Answers1