Use Selenium to wait for progress bar

Question

I'm trying to use Selenium to wait for a progress bar to appear and disappear, which indicates that a search result has started and is then done.

When there is no progress bar, the document looks something like this:

<html lang="en-US">
    <body class="appian-body">
        <a id="skip-to-content"></a>
        <div id="aDiv"></div>
        <div id="anotherOne"></div>
        <div id="OneMore"></div>
        ...
    </body>
</html>

When a search call starts, it looks like this:

<html lang="en-US">
    <body class="appian-body">
        <a id="skip-to-content"></a>
        <div id="aDiv"></div>
        <div id="anotherOne"></div>
        <div id="OneMore"></div>
        ...
        <div id="appian-working-indicator-hidden" style="display: none;"></div>
    </body>
</html>

Note the <div id="appian-working-indicator-hidden" style="display: none;"></div> has appeared, which is also when the progress bar appears. Of course it then goes away when the search is done and then we're back to the original document.

Here are some attempts I've made to wait for this element:

Attempt 1

WebDriverWait wait = new WebDriverWait(driver, 10);
wait.until(ExpectedConditions.visibilityOfElementLocated(By.xpath("/html/body/div[4]")));
wait.until(ExpectedConditions.invisibilityOfElementLocated(By.xpath("/html/body/div[4]")));

Result: Timeout exception on visibilityOfElementLocated (i.e. it's never visible).

Attempt 2

WebDriverWait wait = new WebDriverWait(driver, 10);
wait.until(ExpectedConditions.presenceOfElementLocated(By.xpath("/html/body/div[4]")));
wait.until(ExpectedConditions.invisibilityOfElementLocated(By.xpath("/html/body/div[4]")));

Result: Execution passes through both presenceOfElementLocated and invisibilityOfElementLocated without waiting at all.

So, what else can be done to ensure this element has appeared and disappeared?

Update

I've created these recursive methods:

public void waitForProgressBarToStart() {
    List<WebElement> progressBarList = driver.findElements(By.xpath("//div[@id='appian-working-indicator-hidden']"));
    if(progressBarList.size() == 0) {
        waitForProgressBarToStart();
    } else {
        //for timestamping when the element is found
        DateFormat dateFormat = new SimpleDateFormat("HH:mm:ss");
        Date date = new Date();
        System.out.println(String.format("%s: Progress Bar started", dateFormat.format(date)));
    }
}
public void waitForProgressBarToFinish() {
    List<WebElement> progressBarList = driver.findElements(By.xpath("//div[@id='appian-working-indicator-hidden']"));
    if(progressBarList.size() > 0) {
        waitForProgressBarToFinish();
    } else {
        //for timestamping when the element is gone
        DateFormat dateFormat = new SimpleDateFormat("HH:mm:ss");
        Date date = new Date();
        System.out.println(String.format("%s: Progress Bar done", dateFormat.format(date)));
    }
}
public void waitForResults(){
    waitForProgressBarToStart();
    waitForProgressBarToFinish();
}

After using the timestamps in the methods, I can see that the waitForProgressBarToFinish method takes 10-13 seconds longer than the actual progress bar. waitForProgressBarToStart seems to find the element at the correct time, however.

Why would waitForProgressBarToFinish take longer and how to fix?

score 1 · Answer 1 · answered Mar 21 '20 at 05:28

Since you know that you're specifically looking for a <div> element with an id value of appian-working-indicator-hidden, you should use a more specific xpath query to locate it. I expect that your "Attempt 2" execution was passing through immediately becuase the WebDriver located the 4th <div> on the page, which probably wasn't the div#appian-working-indicator-hidden you're looking for.

You can query for a <div> with a particular id value using the contains() xpath function.

You might rewrite your "Attempt 2" to look something like this:

String xpathQuery = "//div[contains(@id,'appian-working-indicator-hidden')]"

WebDriverWait wait = new WebDriverWait(driver, 10);
wait.until(ExpectedConditions.presenceOfElementLocated(By.xpath(xpathQuery)));
wait.until(ExpectedConditions.invisibilityOfElementLocated(By.xpath(xpathQuery)));

no such luck. I attempted with both presenceOf and visibilityOf. It should work even with the div[4] version, as it's definitely the 4th div — kraftydevil, Mar 21 '20 at 06:55

score 1 · Answer 2 · answered Mar 21 '20 at 08:28

You can fetch the list of that web element, when the element is visible on the page, the size of the list would be 1 and when it is not visible, the size would be 0. So you can check the size of the element to check if it is visible or not.

And i am using a relative xpath for your element instead of the absolute xpath that you are using, because absolute xpaths are very volatile and any minor change in the html structure can have an impact on your script.

Your code should be like:

List<WebElement> progressBarList = driver.findElements(By.xpath("//div[@id='appian-working-indicator-hidden']"));
if(progressBarList.size()>0){
    System.out.println("Element is present");
}
else{
    System.out.println("Element is not present");
}

I've used your solution to create a few methods that appear to find the elements. The problem though, is that they take a very long time. Please see my update and then edit your post here if you have any solutions. ty — kraftydevil, Mar 22 '20 at 05:58

score 1 · Answer 3 · answered Jul 08 '21 at 06:51

For this I use pure JS so that I can escape the built-in waits in Selenium. Negative lookups always been a big pain in the butt when it comes to Selenium

Supplier<Boolean> isLoadingCheck = () -> (Boolean) js.executeScript(
    "return (document.getElementById('nprogress') != null || document.getElementsByClassName('spinner').length > 0)"
);

score 0 · Answer 4 · answered Jun 16 '21 at 17:52

nice logic u used. To prevent the time spilling , u can override your implicit wait time for the second method call and reset it to original value in finally block. I think time would be going in a place when it is searching implicitly for the progress bar when stopped.

Use Selenium to wait for progress bar

Attempt 1

Attempt 2

Update

4 Answers4