Selenium 4 and WebDriver implicitlyWait

Question

In our Selenium projects we use Implicit waits and fluent waits. Basically, we use Implicit waits until we need something explicit. E.g., for normal findElements() calls we depend on the implicit wait, but when waiting for an element to disappear after clicking some button we use a fluent wait. (See code below).

The problem is that lately I'm seeing a number of timeouts when calling implicityWait(n). This seems new. Maybe even in Selenium 4.3. Cannot say for sure, but it has certainly become bothersome. To be clear, "occasionally" means a couple of times over a couple of hundred tests (each calling implicitlyWait dozens of times).

public void waitUntilElementDisappear(By locator, int waitTime) throws Exception, TimeoutException {
    Logger.log(String.format("Waiting [%ds] for [%s] to disappear", waitTime, locator));
    try {
        driver.manage().timeouts().implicitlyWait(Duration.ofSeconds(0));
        WebDriverWait wait = new WebDriverWait(driver, Duration.ofSeconds(waitTime));
        wait.until(ExpectedConditions.invisibilityOfElementLocated(locator));
    } catch (TimeoutException e) {
        Logger.log("Caught timeout exception");
        throw e;
    } catch (Exception e) {
        Logger.log(String.format("Caught exception message:[%s]", e.getMessage()));
        throw e;
    } finally {
        driver.manage().timeouts().implicitlyWait(Duration.ofSeconds(DEFAULT_WAIT));
    }
}

The issue occurs only in the finally block. (DEFAULT_WAIT is usually 180).

Any suggestions?

The exception:

org.openqa.selenium.TimeoutException: java.util.concurrent.TimeoutException
Build info: version: '4.3.0', revision: 'a4995e2c09*'
System info: host: 'DESKTOP-8Q1PEBT', ip: '172.32.6.43', os.name: 'Windows 10', os.arch: 'amd64', os.version: '10.0', java.version: '11.0.15'
Driver info: org.openqa.selenium.remote.RemoteWebDriver
Command: [e8d45a02973053acad1a896d2ed471e6, setTimeout {implicit=0}]
Capabilities {acceptInsecureCerts: false, browserName: chrome, browserVersion: 103.0.5060.134, chrome: {chromedriverVersion: 103.0.5060.134 (8ec6fce403b..., userDataDir: C:\Users\jenkins\AppData\Lo...}, goog:chromeOptions: {debuggerAddress: localhost:59791}, networkConnectionEnabled: false, pageLoadStrategy: normal, platformName: WINDOWS, proxy: Proxy(), se:cdp: ws://172.32.6.43:4444/sessi..., se:cdpVersion: 103.0.5060.134, setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: dismiss and notify, webauthn:extension:credBlob: true, webauthn:extension:largeBlob: true, webauthn:virtualAuthenticators: true}
Session ID: e8d45a02973053acad1a896d2ed471e6
    at org.openqa.selenium.remote.http.netty.NettyHttpHandler.makeCall(NettyHttpHandler.java:65)

I still see following in Chrome 104

10:48:48 Build info: version: '4.3.0', revision: 'a4995e2c09*'
10:48:48 System info: host: 'DESKTOP-8Q1PEBT', ip: '172.32.6.43', os.name: 'Windows 10', os.arch: 'amd64', os.version: '10.0', java.version: '11.0.15'
10:48:48 Driver info: org.openqa.selenium.remote.RemoteWebDriver
10:48:48 Command: [f97c4a4cd7a0d468391badd20e23fe8f, setTimeout {implicit=240000}]
10:48:48 Capabilities {acceptInsecureCerts: false, browserName: chrome, browserVersion: 104.0.5112.81, chrome: {chromedriverVersion: 104.0.5112.79 (3cf3e8c8a07d..., userDataDir: C:\Users\jenkins\AppData\Lo...}, goog:chromeOptions: {debuggerAddress: localhost:56549}, networkConnectionEnabled: false, pageLoadStrategy: normal, platformName: WINDOWS, proxy: Proxy(), se:cdp: ws://172.32.6.43:4444/sessi..., se:cdpVersion: 104.0.5112.81, setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: dismiss and notify, webauthn:extension:credBlob: true, webauthn:extension:largeBlob: true, webauthn:virtualAuthenticators: true}
10:48:48 Session ID: f97c4a4cd7a0d468391badd20e23fe8f]
10:48:48 org.openqa.selenium.TimeoutException: java.util.concurrent.TimeoutException

The implicit wait is global... you don't "call" it, you just set it once. When finding an element Selenium will use it. It should NOT be mixed with any other type of wait. You'll have competing polling loops. (webdriverwait/fluentwait/implicitwait are all polling loops with sleeps inside... default is 1/2 second sleep...) So basically, just use explicit waits. Depending on what the element is, waiting for the element to disappear is problematic as it's not clear when your code will run... (before the item appears?... if it's a loading indicator that might happen) — pcalkins, Aug 02 '22 at 18:41
Also worth noting that an implicit wait does not check for visibility... only presence of. You can read more here: https://www.selenium.dev/documentation/webdriver/waits/#implicit-wait — pcalkins, Aug 02 '22 at 18:43
First, notice in the code I turn off implicit wait (implicitlyWait(Duration.ofSeconds(0)) before instantiating the webdriverwait. When I'm done I turn implicit back on. implicitlyWait(Duration.ofSeconds(DEFAULT_WAIT)); The question is about the timeout I see when turning ON implicit wait. — Rob G, Aug 02 '22 at 21:37
From the doc I see "ExpectedConditions.InvisibilityOfElementLocated Method WebDriver An expectation for checking that an element is either invisible or not present on the DOM." I make this call A LOT. Seems to be responding as expected. Am I missing something? — Rob G, Aug 02 '22 at 21:41
sounds like that should work... the default should be zero, so I guess you're not mixing the two types of waits. Setting the implicitWait time shouldn't be throwing anything, though... it only sets the global timeout period. Can you include the full exception you are receiving? (If it's a "command timeout" you're probably running into a bug in Chrome that's been around a while now... since v99 or so... it tends to happen randomly during a .get() call) — pcalkins, Aug 02 '22 at 22:13
another thing to consider is that I believe the finally block will execute before the throw. (The caught timeout exception is added to the "throw", finally block runs, and thrown exception returned after the finally block...) If that's the case, you should see it reflected in the log before the finally block is called. (Log e.toString() to see full exception) — pcalkins, Aug 02 '22 at 22:49
Checking for invisibility of can be problematic because of timing. You could for instance run the check before the item has ever appeared. (that would be more common for "loading" indicators) Your webdriver call beats the JS to it... or the timeout period might expire before the item has disappeared. Website responsiveness is variable. — pcalkins, Aug 02 '22 at 23:05
For invisibility in my tests the timing is known. E..g., click a button and then wait for the button to disappear. — Rob G, Aug 02 '22 at 23:41
finally happens last before exiting scope. Seems weird with throw but that's how it works. — Rob G, Aug 02 '22 at 23:45
That chrome timeout you mentioned is likely my issue. All recorded failures are chrome. — Rob G, Aug 03 '22 at 01:20
The full exception will mention something about concurrency and specifically a "command timeout". This is the timeout period between the driver and the browser/server... you don't have much control over this unless you are using grid. I think the bug is related to socks calls to chrome dev protocol which Selenium 4 uses for logging. (It's been open for a while now so it sort of remains a mystery.) If you see the command timeout in your log you might try the workaround code mentioned here: https://github.com/SeleniumHQ/selenium/issues/9528#issuecomment-1002776116 — pcalkins, Aug 03 '22 at 17:54
You've left out the exception part. (Just output e.toString()...) — pcalkins, Aug 03 '22 at 20:28
That does look like the Chrome bug. Been trying to find a way around this one for a while now. It has something to do with a promise that's still waiting for a response.... then the next command happens, and you'll see a command timeout thrown. It may also be related to system resources... it's been a thorn in my side for a while now, but seems like a memory leak/threading issue, that's in Chrome Dev Tools somewhere. Doesn't seem to occur in geckodriver. Changing Selenium version or chromedriver version doesn't seem to help. (It's a bug in Chrome/Chromium devtools) — pcalkins, Aug 03 '22 at 21:38
I've been seeing this a lot on .get() method... but it might just be any webdriver method that returns success with data null. — pcalkins, Aug 03 '22 at 21:53
So far this seems to be fixed in Chrome v104. Haven't seen it happening lately. — pcalkins, Aug 08 '22 at 16:20
I still see the timeouts in Chrome 104. I added the latest exception to the question above. — Rob G, Aug 09 '22 at 17:53
That's disappointing. Maybe it's just fixed for .get() calls, or I just haven't seen it yet due to the seemingly random nature of this exception. — pcalkins, Aug 09 '22 at 18:53
one thing you might try is retrying when this exception is caught (don't throw it until a maxretries number is hit) — pcalkins, Aug 10 '22 at 17:29
There is a bit of a problem with the Selenium jar file in that it has duplicate files inside it. This could possibly be the difference between our two builds. One of those duplicated files is part of Netty. You might check to see if your versions of the Netty jars (based on file size since the names will be the same) are the same as I'm using here: https://github.com/petecalkins/Browsermator-Selenium4/tree/master/lib In particular "netty-transport-4.1.78.Final.jar" — pcalkins, Aug 24 '22 at 18:20

score 1 · Answer 1 · answered Aug 31 '22 at 21:49

Still not fixed in ChromeDriver. :(

I decided the only solution was to catch the exception and try again as @pcalkins suggested.

    public void setImplicitWait(int maxWaitTime) {
        try {
            this.driver.manage().timeouts().implicitlyWait(Duration.ofSeconds(maxWaitTime));
        } catch (TimeoutException e) {
            this.driver.manage().timeouts().implicitlyWait(Duration.ofSeconds(maxWaitTime));
        }
    }

Of course there could be a counter and a max retry, but so far I've found just 1 retry and it never fails.

I ended up wrapping a whole bunch of basic methods (e.g., findElement) in the same manner.

I know it is crude but so far seems to be my solution.

undetected Selenium · Answer 2 · 2022-08-02T23:58:19.157

0

To bell the cat as per the Selenium Documentation:

Warning: Do not mix implicit and explicit waits. Doing so can cause unpredictable wait times. For example, setting an implicit wait of 10 seconds and an explicit wait of 15 seconds could cause a timeout to occur after 20 seconds.

Solution

To simulate the expectation for checking for the element to be invisible you can use either of the following approaches:

Using invisibilityOf(WebElement element):

new WebDriverWait(driver, Duration.ofSeconds(waitTime)).until(ExpectedConditions.invisibilityOf(driver.findElement(By.cssSelector("ElementCss"))));

Using invisibilityOfElementLocated(By locator):

new WebDriverWait(driver, Duration.ofSeconds(waitTime)).until(ExpectedConditions.invisibilityOfElementLocated(By.cssSelector("ElementCss")));

edited Aug 02 '22 at 23:58

answered Aug 02 '22 at 23:37

undetected Selenium

183,867
41
278
352

That's why I disable implicit wait first with ```driver.manage().timeouts().implicitlyWait(Duration.ofSeconds(0));``` – Rob G Aug 02 '22 at 23:47
Did you see the code above? ```wait.until(ExpectedConditions.invisibilityOfElementLocated(locator));``` – Rob G Aug 03 '22 at 01:18

Selenium 4 and WebDriver implicitlyWait

2 Answers2

Solution

Linked