Selenium extremely slow on reading the DOM

Question

Selenium interaction with DOM seems extremely slow while doing couple of things in every page instantiation. Throughout the site we have visible spinner that indicates any outstanding API calls resolved or not. In summary I have three methods that make sure the stability of page before performing any action.

1. Check for the DOM ready state
2. Check for any outstanding JQuery calls
3. Check for loading spinners

All of these three are done as a part of the page object instantiation with following methods.

    public static void waitForLoadingAllSpinnersAnywhere(final WebDriver driver){
    final WebDriverWait wait = new WebDriverWait(driver, timeout);
    
    wait.until(waitForDomReadyState());
    wait.until(waitForjQueryToBeInactive());
    List<WebElement> elements = wait.until(ExpectedConditions.presenceOfAllElementsLocatedBy(spinnersLoacator));
    
    for(WebElement element: elements){
        wait.until(invisibilityOfElementLocated(element));  
     }
    }

    private static ExpectedCondition<Boolean> waitForDomReadyState(){

        return new ExpectedCondition<Boolean>() {

            @Override
            public Boolean apply(WebDriver d){

                return ( ((JavascriptExecutor) d).executeScript("return document.readyState;").equals("complete"));
            }
        };
    }


    private static ExpectedCondition<Boolean> waitForjQueryToBeInactive(){

        return new ExpectedCondition<Boolean>() {

            @Override
            public Boolean apply(WebDriver d){

                return (Boolean) ( ((JavascriptExecutor) d).executeScript("return jQuery.active == 0;"));
            }
        };
    }

    public static ExpectedCondition<Boolean> invisibilityOfElementLocated(final WebElement element){

        return new ExpectedCondition<Boolean>() {

            @Override
            public Boolean apply(WebDriver driver){

                try{
                    return !element.isDisplayed();
                } catch (NoSuchElementException | StaleElementReferenceException e){
                    // Returns true because the element is not present in DOM.
                    // The
                    // try block checks if the element is present but is
                    // invisible or stale
                    return true;
                }
            }
        };
    }

Taking an example of a page(say patient page) which has good number of API calls and fetches a lot of data. For a initial class instantiation it takes about 17s(log below). My Selenium knowledge says, the subsequent page instantiation should not take same or more time to check DOM ready state, or JQuery call or spinner waits since there is nothing changing at all. However, every time new page instantiate I see it takes same amount of time taken to check all these three. What's happening there? Does Selenium actually tries to interact with Server every time I do these or just interaction with the client is slow for some reason? If so, what could be the possible answer?

Console log

==== [[Finished waiting for 8 spinner elements found on widget [Patient] after [17] s]]

==== [[Start waiting for 8 spinner elements found on widget [Patient] ]]

==== [[Finished waiting for 8 spinner elements found on widget [Patient] after [17] s]]

==== Browser on [[[Patient]]]

==== [[Start waiting for 8 spinner elements found on widget [Patient] ]]

==== [[Finished waiting for 8 spinner elements found on widget [Patient] after [17] s]]

Environment:

1. Selenium 2.48
2. Firefox 38

I also tried with Selenium 2.52 and firefox 44 with same result

Answer 1

Selenium handles all the waiting on the client side with a request sent to the server for every evaluation until the condition is met. It can quickly degenerate in case of high latency, especially if there are a lot of calls. Moreover some evaluations require a script injection which doesn't help either.

So the best way to improve the performance in your case would be to use a single asynchronous JavaScript call:

public static void waitForLoadingAllSpinnersAnywhere(final WebDriver driver) {
  const String JS_WAIT_SPINNERS = 
      "var callback = arguments[0]; " +
      "(function fn(){ " +
      "  if (document.readyState == 'complete' && jQuery.active == 0) { " +
      "    var elts = $('.spinners'); " +
      "    if (elts.length == 8 && !elts.is(':visible')) " +
      "      return callback(); " +
      "  } " +
      "  setTimeout(fn, 60); " +
      "})();";

   ((JavascriptExecutor)driver).executeAsyncScript(JS_WAIT_SPINNERS);
}

To initialize the timeout:

driver.manage().timeouts().setScriptTimeout(30, TimeUnit.SECONDS);

Answer 2

Your test seems to be all non-native calls and so Firefox should work for you but I am surprised that the Firefox native call to driver.navigate() even worked for you to get to the inital page if you were using 44 and 48. It is well known that 31.6.0 was the last supported native Firefox version. So, I would say you should use Chrome until you figure this out.

But, to answer your thing about slowness. The way you wrote your code, you are highly dependent on jQuery and I would imagine your are having an issue with your calls to jQuery code being delayed, which propagates out to your Selenium test, and further impacted by the fact that your looping through multiple spinners. One thing I have noticed before is that if a page is busy running ajax calls, then your Selenium calls with JavascriptExecutor might have to wait in line for those to give up bits of processor time.

What would I do differently? Well, I would write my spinner waits to operate on the DOM instead of calling JavascriptExecutors to jQuery. Maybe in your case, this is not an option, but I think a well thought out plan can improve the efficiency of your page ready workflow.