Selenium in Python, Unable to Click 'button': Tag not found

Question

TL,DR:

For whatever reason, my selenium python script can't seem to "click" on the buttons I need.

Context:

Hello. My task is one many are probably familiar with: I'd like to automate the process of opening a website, logging in, and clicking on a few drop-down menu links within the website, which will ultimately lead me to a page where I can download a spreadsheet. I'm able to open the web page and log in. To proceed, I have to:

1. Click on the drop down menu header, and
2. in the drop down menu, click on the appropriate option.

Here's a snapshot of the pertinent HTML code from the website:

<td class="x-toolbar-cell" id="ext-gen45">
    <table id="ext-comp-1045" class="x-btn x-btn-noicon" style="width: auto;" cellspacing="0">
        <tbody class="x-btn-small x-btn-icon-small-left">
            <tr>
                <td class="x-btn-ml"><i>&nbsp;</i></td>
                <td class="x-btn-mc">
                    <em class="x-btn-arrow" unselectable="on">
    <button type="button" id="ext-gen46" class=" x-btn-text">Reports</button>
</em>
                </td>
                <td class="x-btn-mr"><i>&nbsp;</i></td>
            </tr>
        </tbody>
    </table>
</td>

The item I need to "click" has a button tag, specifically:

<button type="button" id="ext-gen46" class=" x-btn-text">Reports</button>

To select it with selenium, I've tried the following:

reports_click_element = browser.find_element_by_id('ext-gen46').click()

and when that failed,

reports_element = browser.find_element_by_xpath("//button[contains(text(), 'Reports')]").click()

That one actually executed without an ExceptionMessage error, but I found out it was selecting other elements in the page that had "Reports" text, as opposed to the particular button I need.

When I've tried to zero in on the button I need clicked, the interpreter returned an error message indicating that the html attributes could not be found.

How can I proceed from here? (Should I be focusing on the unselectable="on" tag in the element right above the button I need clicked?)

Please let me know if I can add anything to the question. Thanks in advance.

Update: I have switched into an iframe that I believe the menu is a part of- but I still cannot select the button. So far, here is my Python code:

from selenium import webdriver
from selenium.webdriver.firefox.firefox_binary import FirefoxBinary
import time

binary = FirefoxBinary('C:\Program Files (x86)\Mozilla Firefox\Firefox.exe')
browser = webdriver.Firefox(firefox_binary=binary)

browser.get("https://app.website.com")

login_entry(username, password) # this works fine; it's just a user-created function to login. Ignore.

time.sleep(10) # wait for website's markup to load
browser.switch_to.frame(browser.find_element_by_tag_name("iframe"))
time.sleep(10)

# This is the point where I'm trying to click on the "Reports" button
reports_element = browser.find_element_by_xpath("//*[contains(text(), 'Reports')]") #this refers to other elements
reports_element = browser.find_element_by_xpath("//button[contains(text(), 'Reports')][1]") #no luck here either

Answer 1

A couple of cases which came to mind.

More than one element exists but not all are visible

elements = browser.find_elements_by_xpath("//button[contains(text(), 'Reports')]")
for element in elements:

    if element.is_displayed():
        print "element is visible"
        element.click()
    else:
        print("element is not visible")
        print(element)

The element exists, it would be visible but is out of screen.

from selenium.webdriver.common.action_chains import ActionChains
elements = browser.find_elements_by_xpath("//button[contains(text(), 'Reports')]")
for element in elements:  
    ActionChains(driver).move_to_element(element).perform()
    try:
        element.click()
    except:
        print("couldn't click on {}".format(element)) 

---

Can you also try to record your clicks and keyboard entries with Selenium IDE for Firefox? Then save it as a python script and post it as a comment here?

Answer 2

The "page source" is only what comes with the document request. It will not show any DOM elements created via javascript after the page loads. It does sound like your elements are within some sort of iframe. In the browser console, try this and see if it returns any elements:

document.querySelectorAll('iframe')

EDIT for your update:

Once again, the page source is only what is available at document load. Everything that comes afterward that is loaded dynamically can only be seen by using the browser inspector or by getting parts of the document w/ javascript. The link being basically the same html is probably because it is a link that acts with javascript and isn't meant to lead to an actual html document page. You probably need to do in your code is:

browser.switch_to.frame(browser.find_element_by_css_selector('iframe'))
reports_element = browser.find_element_by_link_text('Reports')
reports_element.click()

Answer 3

I would suggest the following things to try:

1. Switch to the correct frame. There may be a chance of no frame or one frame or more than one nested frames, where your element can be as a child of another. Therefore, you must switch to the right frame before you find the element. My detailed answer is here.

2. Use a more restricted XPATH. For example:

report_element =browser.find_element_by_xpath("//td/em/button[text()='Reports']")

Answer 4

Some ideas:

Print the DOM source of the iframe and check if the html is what you expected:

from selenium import webdriver
from selenium.webdriver.firefox.firefox_binary import FirefoxBinary
import time

binary = FirefoxBinary('C:\Program Files (x86)\Mozilla Firefox\Firefox.exe')
browser = webdriver.Firefox(firefox_binary=binary)

browser.get("https://app.website.com")

login_entry(username, password) # this works fine; it's just a user-created function to login. Ignore.

time.sleep(10) # wait for website's markup to load
browser.switch_to.frame(browser.find_element_by_tag_name("iframe"))
time.sleep(10)

print browser.page_source

It could be, for example, that you switched to the wrong iframe, in case there is more than one in the page.

If the html is correct, then you could try to use the id of the button instead of trying to get it by text:

reports_element = browser.find_element_by_xpath("//button[@id='ext-gen46']")

If that still doesn't work, you can try to click the button with javascript:

browser.execute_script("""
    (function() {
        document.getElementById("ext-gen46").click();
    })()
""")

You can even include jQuery by saving jQuery in a local file, then storing the contents to a variable, and running, like:

with open(JQUERY_PATH) as f:
    jquery = f.read()

browser.execute_script(jquery)

Then you could use it to click the button with:

driver.execute_script("""
    (function() {
        jQuery(document).ready(function($) {
            $('#ext-gen46').click();
        }, jQuery)
    })()
""")

Answer 5

I'm doing something similar with Siebel OneView, which puts most of its control buttons and menus inside Java screen elements. These elements prevent me from finding the HTML objects to activate with Selenium, so I have fallen back to using pyautogui:

import pyautogui
import time
#look for and click the Activites menu
try:
    x,y = pyautogui.locateCenterOnScreen('acts_button.png')
    try:
        pyautogui.click(x, y)
    except PermissionError:
        #allow load - note this is AFTER the click permission error
        time.sleep(7)
        pass
except TypeError:
    <code to deal with NOT finding the button>

This looks for a copy of a previously taken screenshot, stored in the same file location as the python script. In this section the image is called acts_button.png.

[Quick note: There are two try: except: statements in here. The inner one deals with any problems clicking the button as Windows will often throw out permission errors. The outer one is more important and tells your script what to do if it can't find the button. In my code, I try clicking on a preset x,y location with pyautogui.click(958, 169); if that fails, I ask the user for input (Failures detected by the next screen not loading).

The screenshots themselves are created by using commands like this

 acts_button = pyautogui.screenshot('acts_button.png', region=(928,162,63,15))

Where the region is a tuple of the following parts of your target button, measured as pixels

• left hand edge
• top edge
• width
• height

Now all you need is a way to find the (x,y) values for when you take the screenshot. Fortunately I have code for that too:

#! python3
import pyautogui, sys, time
print('Press Ctrl-C to quit.')
try:
    while True:
        x, y = pyautogui.position()
        positionStr = 'X: ' + str(x).rjust(4) + ' Y: ' + str(y).rjust(4)
        print(positionStr, end='')
        print('\b' * len(positionStr), end='', flush=True)
        time.sleep(1)        
except KeyboardInterrupt:
    print('\n')

This will pop up a small window with the (x,y) coordinates of the mouse, updating once per second.

The routine to set up a new click event is therefore

1. Open the site you want to navigate
2. Use the mouse locator script to find the settings for the screenshot tuple
3. Use pyautogui.screenshot to save an image of the button
4. Use the main script to look for and click that image

After a couple of goes you should be able to set up a new click event in less than a minute. You can either use this method where you know Selenium will fail, or as a way of catching and managing exceptions when Selenium might fail.

Answer 6

You should check whether there is only one iframe on current page. You can use

len(browser.find_elements_by_tag_name('iframe'))

If output is greater than 1 and you use browser.switch_to.frame(browser.find_element_by_tag_name("iframe")), it means that you're trying to switch to first iframe found by webdriver, but you actually might need to switch to another iframe

Good solution is to find required iframe in button's ancestors with F12 or right-click on button + inspect element and use specific attributes of iframe in XPath that will match exact iframe, e.g.

browser.switch_to.frame(browser.find_element_by_xpath('//iframe[@class="iframe_class_name"]'))

Bad solution is to define index of required iframe by exhaustive search like:

browser.switch_to.frame(browser.find_elements_by_tag_name("iframe")[0])
reports_element = browser.find_element_by_xpath("//button[text()='Reports'][contains(@id, 'ext-gen')]")
reports_element.click()
...
browser.switch_to.frame(browser.find_elements_by_tag_name("iframe")[1])
reports_element = browser.find_element_by_xpath("//button[text()='Reports'][contains(@id, 'ext-gen')]")
reports_element.click()
....

Answer 7

Why not use python request (with session) and BeautifulSoup modules to do this kind of job (user interact on a website) ?