Selenium find_element throwing exception even though the element exists

Question

My code:

soup = BeautifulSoup(driver.page_source,features="html.parser")
applications_domains = []

for card in soup.find_all("div", {"class":"ant-row"}):
    for url in card.find_all("a"):
    applications_domains.append(url.get("href"))

for aplications_domain in aplication_domains:
    try:
        WebDriverWait(driver,10).until(EC.presence_of_element_located((By.XPATH,"//a[@href='" + 
applications_domain + "']")))
        driver.find_element_by_xpath("//a[@href='" + applications_domain + "']").click()
    except:
        soup = BeautifulSoup(driver.page_source,features="html.parser")
        print(soup.find_all("a",{"href":applications_domain}))
        print(f"test error {applications_domain}")
        print("-----------------------")

I have an issue with find_element_by_xpath not finding the element even though it exists. I double checked using soup if indeed the element exists and it does as per output.

Output:

<a href="applications_domain"><b></b></a>
test error applications_domain

I have a loop that goes through each application domain (contains data from each href) however, it finds and clicks on the a href element most of the time but does not for some and I have no idea why.

Here is the site html. There are many div id="application_name_list" and each contain different a href that I need to click through

<div class="ant-row" style="margin-left: -6px; margin-right: -6px;">
<div id="application_name_list" class="ant-col-8 dyff-home-app-search-result-item" style="padding-left: 6px; padding-right: 6px;">
    <a href="/dyfflaunch/domain/gco/app/di_data_customer_experience_conversation_processor/features">di_data_customer_experience_conversation_processor<b></b></a>
</div>
<div id="application_name_list" class="ant-col-8 dyff-home-app-search-result-item" style="padding-left: 6px; padding-right: 6px;">
    <a href="/dyfflaunch/domain/gco/app/di_kafka_configservice_agentqueuegroup_dim_v1-prod/features">di_kafka_configservice_agentqueuegroup_dim_v1-prod<b></b></a>
</div>
<div id="application_name_list" class="ant-col-8 dyff-home-app-search-result-item" style="padding-left: 6px; padding-right: 6px;">
    <a href="/dyfflaunch/domain/gco/app/di_kafka_configservice_phoneinventory_dim_v1-prod/features">di_kafka_configservice_phoneinventory_dim_v1-prod<b></b></a>
</div>
</div>
enter code here

Is your code the exact one you ran or just the mock up? I see the for loop is not correct. You are appending urls in "applications_domains" list and you are looping through "aplication_domains". — Azhar Syed, Apr 08 '20 at 12:29

score 0 · Answer 1 · answered Apr 08 '20 at 12:54

0

This is one, pretty generic way of doing it:

a_tags=driver.find_elements_by_xpath("//div[@id='application_name_list']//a")

for a_tag in a_tags:
    a_tag.click()

If you have examples where this doesn't work, please add one to the question.

answered Apr 08 '20 at 12:54

0buz

3,443
2
8
29

added 3 examples that it is not working for. Thank you for pointing it out – Roitko Apr 08 '20 at 13:11
If you compare the structure for one that works and one that doesn't, is there anything different whatsoever? One thing that comes to mind for example, is whether you have an `iframe` somewhere higher above `
`.
– 0buz Apr 08 '20 at 14:01
1

I checked whole html and there is no `iframe`. I tried comparing the htmls before submitting this but they have the same structure. Here is one working as an example. `
di_kafka_configservice_agentqueuegroup_dim_v1-prod
` – Roitko Apr 08 '20 at 14:08
I see the working example has a white space as text() value for the `a` node, but I'd be extremely surprised if that's what makes the difference. I see the `div`s are the result of a search, correct? Are some of these results outside the visible screen i.e. do you need to scroll to reach them? And if so, are these (the ones outside the visible screen) the results which fail? If that is the case, use `ActionChains` to bring the element in focus: `ActionChains(driver).move_to_element(a_tag).click().perform()`. – 0buz Apr 08 '20 at 14:26
these examples are on the top of the site. Nevertheless I will try to use this code instead and see what happens. Will let you know if it worked or not. – Roitko Apr 08 '20 at 14:35
I am still receiving the same errors. I will try to use text of the `a href` tag instead of `href` value and see if it helps. – Roitko Apr 08 '20 at 14:50
I found out what was causing the issue. The only thing that was different from the working links was the length of text. After further testing I found out that links are overlapping on the screen. See https://prnt.sc/rvhumn for more information. Do you know by any chance how this could be fixed? – Roitko Apr 08 '20 at 15:05
1

Found the solution [link](https://stackoverflow.com/questions/37879010/selenium-debugging-element-is-not-clickable-at-point-x-y/37880313) – Roitko Apr 08 '20 at 15:17

score 0 · Answer 2 · answered Apr 08 '20 at 13:00

I would suggest use WebDriverWait() and wait for visibility_of_all_elements_located() and then use following css selector to click.

driver.get("url here")
WebDriverWait(driver,10).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR,".ant-row")))
for link in driver.find_elements_by_css_selector(".ant-row>#application_name_list>a[href]"):
    link.click()

If you want to use beautiful soup and selenium to do that then try this one.

driver.get("url here")
WebDriverWait(driver,10).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR,".ant-row")))

soup = BeautifulSoup(driver.page_source,features="html.parser")
applications_domains = []

for url in soup.select(".ant-row>#application_name_list>a[href]"):
    applications_domains.append(url['href'])

for applications_domain in applications_domains:
    try:
        WebDriverWait(driver,10).until(EC.visibility_of_element_located((By.XPATH,"//a[@href='" + applications_domain + "']")))
        driver.find_element_by_xpath("//a[@href='" + applications_domain + "']").click()
    except:
        soup = BeautifulSoup(driver.page_source,features="html.parser")
        print(soup.find_all("a",{"href":applications_domain}))
        print("test error {applications_domain}")
        print("-----------------------")

what bothers me the most that the `a href` is there as it is in `applications_domains` list as well as it is find-able using `soup`. I will try to do another `WebDriverWait`. — Roitko, Apr 08 '20 at 13:06
@Roitko : The sample html which you have posted is working fine to me.If it possible to share url then I would be in a better position to know why this isn't working for you.Thanks. — KunduK, Apr 08 '20 at 13:53
unfortunately I am not able to share the HTML as it is working only on extranet. However, thank you for looking into this! — Roitko, Apr 08 '20 at 14:00

score 0 · Accepted Answer · answered Apr 08 '20 at 15:19

0

The issue was caused by overlapping and solved as per Solution Error message returned was selenium.common.exceptions.ElementClickInterceptedException: Message: element click intercepted: Element is not clickable at pointbut due to my poor knowledge of error handling the error was not shown as expected. Thank you all for help!

answered Apr 08 '20 at 15:19

Roitko

117
1
8

you would have mentioned that error in your original post.It would have resolved two hours before. – KunduK Apr 08 '20 at 15:26
@KunduK I was not aware of the error as I did not output it. It was my beginner mistake and I apologize for wasting everyone's time! – Roitko Apr 08 '20 at 15:37
I am not harsh.I hope you understood what i mean.Its lesson learned always provide more information like if you are getting any error which helps OP to provide resolution towards problem.All the best.Happy Learning. – KunduK Apr 08 '20 at 16:02

Selenium find_element throwing exception even though the element exists

3 Answers3