Python Selenium Explicit Wait

Question

I'm having a lot of troubles to find a clean and reliable way to wait in my Python Selenium tests. I've been using Implicit wait for a long time and for some reason it started being unstable so I switched to explicit wait. For some reason, I can't figure out a way for my tests to work 100% of the time. Right now, even though the code being tested and the test itself does not change, problems are happening again and again. (I'd say it works 50% of the time). When problems are happening, it's usually and the same lines. Here are some of them :

for th in td_list:
    if(th.text == "18"):
        th.click()
        break
    time.sleep(3)
    element = WebDriverWait(driver, 20).until(
      EC.element_to_be_clickable((By.ID, "closeModalDisplayBill"))
    )

Here the "closeModalDisplayBill" is a button that has to be pushed. The problem it raises is the following :

Traceback (most recent call last):
  File "./scripts/test-sl/SeleniumTest.py", line 54, in test_selenium_launcher
    frontAdminErr = SlFrontAdminErr(driver, self.add)
  File "/Users/JNEB/WebstormProjects/backend/backend/scripts/test-sl/SlFrontAdminErr.py", line 45, in __init__
    driver.find_element_by_id("closeModalAddComp").click()
  File "/Library/Python/2.7/site-packages/selenium/webdriver/remote/webelement.py", line 78, in click
    self._execute(Command.CLICK_ELEMENT)
  File "/Library/Python/2.7/site-packages/selenium/webdriver/remote/webelement.py", line 499, in _execute
    return self._parent.execute(command, params)
  File "/Library/Python/2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 297, in execute
    self.error_handler.check_response(response)
  File "/Library/Python/2.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 194, in check_response
    raise exception_class(message, screen, stacktrace)
ElementNotInteractableException: Message:

Another place where problems are recurrent is when Selenium try to open a modal to edit some stuff (clicking or clearing inputs is pretty unstable). Here is the code :

time.sleep(5)
driver.implicitly_wait(15)
driver.find_element_by_id("btnLogin").click()
time.sleep(1)
driver.find_element_by_id("login-email").clear()

And here is the log when the bug happen (trying to clear the login-email input field) :

Traceback (most recent call last):
  File "./scripts/test-sl/SeleniumTest.py", line 39, in test_selenium_launcher
    frontUserTest = SlFrontUser(self, driver, self.add)
  File "/Users/JNEB/WebstormProjects/backend/backend/scripts/test-sl/SlFrontUser.py", line 21, in __init__
    driver.find_element_by_id("login-email").clear()
  File "/Library/Python/2.7/site-packages/selenium/webdriver/remote/webelement.py", line 93, in clear
    self._execute(Command.CLEAR_ELEMENT)
  File "/Library/Python/2.7/site-packages/selenium/webdriver/remote/webelement.py", line 499, in _execute
    return self._parent.execute(command, params)
  File "/Library/Python/2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 297, in execute
    self.error_handler.check_response(response)
  File "/Library/Python/2.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 194, in check_response
    raise exception_class(message, screen, stacktrace)
InvalidElementStateException: Message: Element is not currently interactable and may not be manipulated

The same test can be successful four or five times in a row and then bug the same amount of time. (Everything here is tested from my own computer). When those tests are done through Jenkins it gets worse. Some of the time the full suite of test can be done in 10 or 8 minutes, but some of the time thoses tests will be done/failing after 30 minutes. I figure that the slow factor on Jenkins can be one of the reason my tests are unstable, but that doesn't explain why those bug appear regularly on my own computer.

All those tests are launched from another Python Script that initiate the firefox driver instance and then launch all the tests like such :

class SeleniumTestLauncher(unittest.TestCase):
    environmnent = "envName"
    add = ""
    quickbooks_url = "https://developer.intuit.com/"
    port = ""
    driver = webdriver.Firefox()
    base_url = ""

    def setUpAdd(self):
        self.driver.implicitly_wait(30)
        self.verificationErrors = []
        self.accept_next_alert = True

    def test_selenium_launcher(self):
        driver = self.driver
        ### -- Here just call every selenium test -- ###

        ## -- Test Front User -- ##
        frontUserTest = SlFrontUser(self, driver, self.add)

SECOND EDIT : As suggested I removed all the implicitly_wait() for explicit_wait. Seems a bit more stable but I still encounters bugs. For instance :

element = WebDriverWait(driver, 20).until(
      EC.element_to_be_clickable((By.ID, "closeModalDisplayBill"))
)
driver.find_element_by_id("closeModalDisplayBill").click()

Even though the driver should wait for 'closeModalDisplayBill' to be clickable to actually attempt to click, I get this error :

Traceback (most recent call last):
  File "./scripts/test-sl/SeleniumTest.py", line 53, in test_selenium_launcher
    frontUserTest = SlFrontUser(self, driver, self.add)
  File "/Users/JNEB/WebstormProjects/backend/backend/scripts/test-sl/SlFrontUser.py", line 37, in __init__
    driver.find_element_by_id("closeModalDisplayBill").click()
  File "/Library/Python/2.7/site-packages/selenium/webdriver/remote/webelement.py", line 78, in click
    self._execute(Command.CLICK_ELEMENT)
  File "/Library/Python/2.7/site-packages/selenium/webdriver/remote/webelement.py", line 499, in _execute
    return self._parent.execute(command, params)
  File "/Library/Python/2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 297, in execute
    self.error_handler.check_response(response)
  File "/Library/Python/2.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 194, in check_response
    raise exception_class(message, screen, stacktrace)
ElementNotInteractableException: Message:

EDIT 3 : Using only explicit wait is as unstable as before. Most of the errors I run into are those : "InvalidElementStateException: Message: Element is not currently interactable and may not be manipulated". I don't understand why Selenium tries to click on the "input field" even though I use an explicit_wait. For instance :

time.sleep(3)
element = WebDriverWait(driver, 20).until(      
  EC.presence_of_element_located((By.ID, "login-email"))
)
driver.find_element_by_id("login-email").clear()

Here I ask selenium to wait for 'login-email' to be present, and if it is, to click on it. The wait returns before the 20 seconds timeout but the .clear() function throw a "InvalidElementStateException: Message: Element is not currently interactable and may not be manipulated".

What browser are yo using? Is it consistent across different browsers? — Levi Noecker, Oct 22 '17 at 12:37
@LeviNoecker I'm always using Firefox. I'm editing the post with the init of the webdriver — jineb92, Oct 22 '17 at 12:41
@jineb92 Mixing implicit and expicit waits is not recommended as you will get unpredictable timeouts. — Grasshopper, Oct 22 '17 at 13:01
@Grasshopper By mixing implicit & explicit timeout do you mean that I should use "time.sleep(x)" or "implicitly_wait()" ? If I choose to use explicit timeout at some point, I should change all of the implicit timeout to explicit timeouts ? — jineb92, Oct 22 '17 at 13:03
@jineb92 Use either explicit or implicit wait. Does the code work if you do not use implicit wait at all? — Grasshopper, Oct 22 '17 at 13:44
@Grasshopper Well I never tried to not use implicit wait at all but I'm pretty sure it won't work. I removed all the 'implicitly_wait(x)' call from the code and changed them for explicit_wait. Do you think I should also remove all the 'time.sleep()' ? — jineb92, Oct 22 '17 at 13:47
`sleep()` is a bad practice and implicit waits are not recommended. Mixing implicit and explicit waits are warned against in the Selenium docs. I would remove all `sleep()`s and implicit waits and then add explicit waits as needed. — JeffC, Oct 22 '17 at 19:01

krokodilko · Accepted Answer · 2017-10-22T19:32:44.327

closeModalDialog suggests that your application is based on some Ajax framework.

Automating Ajax applications using Selenium can sometimes be a big challenge. An element seems to be visible and clickable at time X when Webriver is trying to locate the element, but there are still some pending javascript code which modifies the DOM and will change this element at time X+10 miliseconds when the test is trying to click on the element, and Webdriver in this case usually throws one of: ElementNotVisibleException, InvalidElementStateException, StaleElementReferenceException, ElementNotInteractableException.
The test is working fine on a local (fast) machine, but fails on much slower virtual machines, or when the application is under heavy load and it's response times increase.

First of all discuss this issue with your developers.
Our appliaction is based on JSF + Primefaces framework and we had many issues like above while trying to automate our product. We have discussed the issues with our developers, and they put simple Status component in the header on each page, which indicates whether there are ongoing (active) ajax request or not. Secondly, since Primefaces is using jQuery (many ajax framoworks are using jQuery), we are also checking if jQuery status is ready (for details see this answer
We have created a simple waitForPrimefacesAndJQuery function that waits unitil Primefaces status and jQuery status are ready, and have been using this function everywhere alongsize with standard WebDriverWait.
This has solved 99,9% of issues.

There are still some components which don't want to cooperate though.
For them we usually use a brute-force method - simply cath the exception and try again a few times, for example in this way:

for(int i=0; i <= 5; i++ ){
  try{
    findElement( element ).click();
    break;
  } catch( ElementNotVisibleException | InvalidElementStateException 
          | StaleElementReferenceException | ElementNotInteractableException )
  {
    // swallow the exception, wait some time and try again
    sleep( 500 );
  }
}

Thanks a lot for this answer, I guess I'll implement the brute force method for the recurrent problems. We are not using jQuery but Angular/Type Script but I feel like the problem in jQuery is the one I have. Problem happens only when I run the test on slow machine. Thanks again — jineb92, Oct 23 '17 at 07:51

Python Selenium Explicit Wait

1 Answers1