Find Element doesn't always succeed. Can this be improved?

Question

I have a certain script which refers to certain elements in my tested UI. some of the elements are located by their ID and some by their Xpath. This script collects some information from my UI after a certain scenario. I tun this script repeatedly in order to find a specific result.

Most of the time all the elements are found. It happens that after about 100 (sometimes after several tens of cycles) cycles the script stops and most of the time it is related to an element that wasn't found. I understand that this kind of automation failure is common, I also assume that sometimes it might be related to a certain issue related to my UI (or even to the device that is managed by the UI).

Is it possible/recommended to design my automation in a way that will be more tolerant to certain situations (by the way I did use a built-in implicit wait) ?

Is it recommended in these king of tests to create retry mechanisms ?

Here is the script:

from selenium import webdriver
from datetime import datetime
import time


def main():
    i = 0
    while True:
        i = i +1
        execute_code(i)

        if i == 500:
            break

def Cold_Restart(browser):

    browser.switch_to.default_content()

    browser.switch_to.frame('box_menu')
    browser.switch_to.frame('box_menu')
    SysBtn = browser.find_element_by_id('System')
    SysBtn.click()

    browser.switch_to.default_content()

    browser.switch_to.frame('main_menu')
    Mainten_Btn = browser.find_element_by_id('Maintenance')
    Mainten_Btn.click()

    browser.switch_to.default_content()

    browser.switch_to.frame('main_body')
    Mntn_Rst_Tab = browser.find_element_by_id('tab_restart')
    Mntn_Rst_Tab.click()

    browser.switch_to.frame('maint_sys')
    Cold_Rst_Btn = browser.find_element_by_id('cold_restart')
    Cold_Rst_Btn.click()

    #In order to confirm the Alert Message I first need to switch to the alert pop-up message and then accept it
    alertDialog = browser.switch_to_alert()
    alertDialog.accept()

    time.sleep(205)

    return


def Port_Admin_Down(browser):

    browser.switch_to.default_content()

    browser.switch_to.frame('box_menu')
    browser.switch_to.frame('box_menu')
    Port_19 = browser.find_element_by_id('Port-19')
    Port_19.click()


    browser.switch_to.default_content()

    # Show the Port Configuration TAB
    browser.switch_to.frame('main_body')
    CFP2_Info = browser.find_element_by_id('tab_general')
    CFP2_Info.click()

    browser.switch_to.frame('config_port') # Move to the inner frame that holds the configuration fields and buttons

    Admin_Down_Btn = browser.find_element_by_id('red_button')
    Admin_Down_Btn.click()

    #In order to confirm the Alert Message I first need to switch to the alert pop-up message and then accept it
    alertDialog = browser.switch_to_alert()
    alertDialog.accept()

    time.sleep(5)
    return


def Port_Admin_Up(browser):

    browser.switch_to.default_content()

    browser.switch_to.default_content()

    browser.switch_to.frame('box_menu')
    browser.switch_to.frame('box_menu')
    Port_19 = browser.find_element_by_id('Port-19')
    Port_19.click()
    browser.switch_to.default_content()

    # Show the Port Configuration TAB
    browser.switch_to.frame('main_body')
    CFP2_Info = browser.find_element_by_id('tab_general')
    CFP2_Info.click()

    browser.switch_to.frame('config_port') # Move to the inner frame that holds the configuration fields and buttons

    Admin_Down_Btn = browser.find_element_by_id('green_button')
    Admin_Down_Btn.click()

    #In order to confirm the Alert Message I first need to switch to the alert pop-up message and then accept it
    alertDialog = browser.switch_to_alert()
    alertDialog.accept()    

    time.sleep(5)
    return

def Gui_Login(browser, URL):
    browser.get(URL)
#    browser.implicitly_wait(20) # Implicit wait


    browser.maximize_window()

    # Find the User Name text box and fill the User name
    user_name_box = browser.find_element_by_id('u_name_box')
    user_name_box.click()
    user_name_box.send_keys('admin')

    # Find the Password text box and fill the Password
    user_pass_box = browser.find_element_by_id('u_pass_box')
    user_pass_box.click()
    user_pass_box.send_keys('admin')

    #webdriver.ActionChains(driver).send_keys(Keys.ESCAPE).perform()

    login_button = browser.find_element_by_id('login_but')
    login_button.click()

    return

def Gui_Logout(browser):

    browser.switch_to.default_content() # Get back to the default starting point


    # In order to click the Logout button I needed to pass through two frames
    browser.switch_to.frame('box_menu')
    browser.switch_to.frame('box_menu')
    logout_btn = browser.find_element_by_id('Logout')
    logout_btn.click()

    return

def Sample_Uplink(browser):

    browser.switch_to.default_content()

    # Go to the Configuration Pane (main_menu)
    browser.switch_to.frame('main_menu')
    Configuration_Btn = browser.find_element_by_id('Configuration')
    Configuration_Btn.click()

    browser.switch_to.default_content()

    # Go to the Uplink 1 CFP2 information and take the Rx Pwr
    browser.switch_to.frame('box_menu')
    browser.switch_to.frame('box_menu')
    Port_19 = browser.find_element_by_id('Port-19')
    Port_19.click()

    browser.switch_to.default_content()

    # Show the Optic Module information TAB
    browser.switch_to.frame('main_body')
    CFP2_Info = browser.find_element_by_id('tab_XFP')
    CFP2_Info.click()

    # Collect the Rx Pwr from the CFP2 Info screen
    browser.switch_to.frame('config_port') # Move to the inner frame that holds all the tables
    #browser.find_element_by_class_name('table_round_corner')
    Rx_Pwr = browser.find_element_by_xpath('html/body/form/div[1]/div/table/tbody/tr[2]/td[2]') # Take the Rx Pwr according to its Xpath
    # print (Rx_Pwr.text) # print the Rx Pwr result to screen
    RcvPwr = Rx_Pwr.text

    # Collect the OSNR measurement from the CFP2 Info screen
    OSNR = browser.find_element_by_xpath('html/body/form/div[1]/div/table/tbody/tr[4]/td[2]')
    OSNR_Lvl = OSNR.text


#    time.sleep(5)
    return (RcvPwr, OSNR_Lvl)



def Save_2_File(Rx_Pwr, OSNR_Lvl, i):
    file = open("test_results.txt", "a")
    file.write("%i. " %i)
    file.write(datetime.now().strftime('%H:%M:%S %d-%m-%Y    ')) # Print Time & Date to the text file 
    file.write(Rx_Pwr) # Print the Rx_Pwr to the text file
    file.write('%10s' %(OSNR_Lvl)) # Format the placement of the OSNR value
    file.write('\n') # Make sure that the next iteration will write the results in the next line
    file.close() # Closing the file
    return


def execute_code(i):
    profile = webdriver.FirefoxProfile()
    profile.accept_untrusted_certs = True

    browser = webdriver.Firefox(firefox_profile = profile)
    browser.implicitly_wait(20) # Implicit wait


    URL1 = 'http://10.0.1.131' #  First device that will be Cold Restarted
    Gui_Login(browser, URL1)

    Cold_Restart(browser)
    (RcvPwr, OSNR_Lvl) = Sample_Uplink(browser)
    Save_2_File(RcvPwr, OSNR_Lvl, i)

    URL2 = 'http://10.0.1.134'
    Gui_Login(browser, URL2)

    Cold_Restart(browser)
    (RcvPwr, OSNR_Lvl) = Sample_Uplink(browser)
    Save_2_File(RcvPwr, OSNR_Lvl, i)


    browser.get(URL1)

    Port_Admin_Down(browser)
    Port_Admin_Up(browser)

    time.sleep(5)

    browser.get(URL2)   

    #Get Rx Pwr and OSNR and save in file
    (RcvPwr, OSNR_Lvl) = Sample_Uplink(browser)
    Save_2_File(RcvPwr, OSNR_Lvl, i)


    Port_Admin_Down(browser)
    Port_Admin_Up(browser)

    time.sleep(5)

    browser.get(URL1)   

    #Get Rx Pwr and OSNR and save in file
    (RcvPwr, OSNR_Lvl) = Sample_Uplink(browser)
    Save_2_File(RcvPwr, OSNR_Lvl, i)


    browser.quit()


if __name__ == '__main__':
    main()

We use the flaky nose plugin for re-running failed integration tests. Might be useful for you too. https://pypi.python.org/pypi/flaky — BoboDarph, Jul 13 '17 at 10:18

score 1 · Answer 1 · answered Jul 13 '17 at 10:22

If you want to avoid error in some loops. try/catch will help.

First: Move some code to init/quit webdriver from execute_code() to main, and then try/catch the execute_code().

def main():
    i = 0
    while True:
        i = i +1
        profile = webdriver.FirefoxProfile()
        profile.accept_untrusted_certs = True
        browser = webdriver.Firefox(firefox_profile = profile)
        browser.implicitly_wait(20) # Implicit wait
        try:
            execute_code(i, browser)
            browser.quit()
            if i == 500:
                break
        except:
            browser.quit()

Second: Modify execute_code(). Add browser as a parameter. Remove the code to init/quit webdriver.

def execute_code(i, browser):
    URL1 = 'http://10.0.1.131' #  First device that will be Cold Restarted
    Gui_Login(browser, URL1)

    Cold_Restart(browser)
    (RcvPwr, OSNR_Lvl) = Sample_Uplink(browser)
    Save_2_File(RcvPwr, OSNR_Lvl, i)

    URL2 = 'http://10.0.1.134'
    Gui_Login(browser, URL2)

    Cold_Restart(browser)
    (RcvPwr, OSNR_Lvl) = Sample_Uplink(browser)
    Save_2_File(RcvPwr, OSNR_Lvl, i)


    browser.get(URL1)

    Port_Admin_Down(browser)
    Port_Admin_Up(browser)

    time.sleep(5)

    browser.get(URL2)   

    #Get Rx Pwr and OSNR and save in file
    (RcvPwr, OSNR_Lvl) = Sample_Uplink(browser)
    Save_2_File(RcvPwr, OSNR_Lvl, i)


    Port_Admin_Down(browser)
    Port_Admin_Up(browser)

    time.sleep(5)

    browser.get(URL1)   

    #Get Rx Pwr and OSNR and save in file
    (RcvPwr, OSNR_Lvl) = Sample_Uplink(browser)
    Save_2_File(RcvPwr, OSNR_Lvl, i)

Can you please explain why do you think that your modifications will do the required effect ? — Moshe S., Jul 13 '17 at 10:26
@MosheS. If the element takes too long time to display, you can use `wait`. However, if the element is not displayed because of an error, `wait` won't help. You need try/catch to cover all the statements. Otherwise, you need to create your own extension methods for `find_element` to catch the error. — Buaban, Jul 13 '17 at 10:40
If I understand your implementation right, when I will run into a cycle that a certain element won't be found my automation won't be interrupted, instead it will continue. — Moshe S., Jul 13 '17 at 12:45
@MosheS. correct. When exceptions are expected behavior, we need try/catch. You can choose to try/catch the whole block of code, or create extension methods and add try/catch inside the methods. If you are interested in the extension, I can provide you another answer. — Buaban, Jul 14 '17 at 03:35
I am very interested in the extension methods. I would appreciate it very much. I also believe that other beginners like me would learn from this kind of example a lot — Moshe S., Jul 15 '17 at 05:08

Arount · Accepted Answer · 2017-07-13T11:00:43.507

1

I would wait for expected elements and except a NoSuchElementException if it fails.

try:
    element = WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.ID, "Maintenance"))
    )
except NoSuchElementException:
    # Handle your error case here
else:
    element.click()

At this point you can stop gracefully, retry the last call, just skip this test, etc...

Check python selenium waits doc

Edit for comments

I'm not big fan of this, but if adding some lines bother you, you can always make a kind of wrapper for that:

def wait_for_func(driver, element_tuple, func_success, func_failed, time=10):
    try:
        element = WebDriverWait(driver, time).until(
            EC.presence_of_element_located(element_tuple)
        )
    except NoSuchElementException:
        return None, func_failed(element)
    else:
        return element, func_success(element)

def do_click(element):
    return element.click()

def do_print():
     return 'Something gone wrong'

element, result = wait_for_func(driver, (By.ID, 'Maintenance'), func_success=do_click, func_failed=do_print)

This will avoid to repeat try/except everywhere, but you surelly know Zen of Python:

Explicit is better than implicit. Simple is better than complex. Complex is better than complicated.

Anyway, if you do something like that you could notice it's not that great to define a function only for clicking an element, so lambda can help you:

wait_for_func(driver, (By.ID, 'Maintenance'), lambda el: el.click(), do_print)

I find lambda is not that beautiful, but as you wish.

edited Jul 13 '17 at 11:00

answered Jul 13 '17 at 10:26

Arount

9,853
1
30
43

By doing it so I need to repeat this method to any of the elements I refer to in my automation. Right? It looks like too much overhead, is it the conventional way ? – Moshe S. Jul 13 '17 at 10:39
I am not confident because I am new to Python. I am not familiar with any of the right conventions of doing the code right. I thank you very much for your answer and I will try it. The only extra that I wanted is to know if it is "right" to use the try catch to any time that I search for (and want to do actions) an element. By the way, sorry but can you please explain the "lambda el: el.click()" ? – Moshe S. Jul 13 '17 at 11:16
Yes, try / except is **always** what you shoul prefere from every other possibility. This even have a name: "Easier to ask for forgiveness than permission" ([EAFP](https://docs.python.org/2/glossary.html#term-eafp)) – Arount Jul 13 '17 at 11:27
can you please explain the "lambda el: el.click()" ? Why shouldn't I use this convention ? – Moshe S. Jul 13 '17 at 12:50
1

[python lambda](http://www.secnetix.de/olli/Python/lambda_functions.hawk), [Why lambda are useful](https://stackoverflow.com/questions/890128/why-are-python-lambdas-useful). In two words, lambda is a way to **define** a function without **declaring** it. It avoid to create a function if you only needs it once. It's not that great because [Readability counts.](https://www.python.org/dev/peps/pep-0020/#id3) :) – Arount Jul 13 '17 at 12:53
To sum it all up, by using lambda, as you suggested, I will utilize the special wait_fo_element function to other actions than click() ? – Moshe S. Jul 13 '17 at 13:56
exacte. Could we remove all our non useful comments here? It's less ugly :) - I let the EAFP & lambda one, keep this one for some minutes and delete if after – Arount Jul 13 '17 at 13:58

Find Element doesn't always succeed. Can this be improved?

2 Answers2

Edit for comments