Selenium tests fail with azure pipeline sometimes and pass locally

Question

Test cases fail while not finding a link or any validation messages after a button click.

I used explicit waits for page to load:

var waitForDocumentReady = new WebDriverWait(WebDriver, TimeSpan.FromSeconds(10));
            waitForDocumentReady.Until((wdriver) => (WebDriver as IJavaScriptExecutor).ExecuteScript("return document.readyState").Equals("complete"));

For wait specific div of validation messages:

 WebDriverWait wait = new WebDriverWait(WebDriver, TimeSpan.FromSeconds(10));
            wait.Until(SeleniumExtras.WaitHelpers.ExpectedConditions.ElementExists(By.ClassName("validationErrors")));

But test cases pass sometimes with pipeline and fails sometimes.

How are you running your test cases in azure? Headless, or are you rendering a browser? I have seen issues with unexplained failures like this when running in non-headless mode. — CEH, Nov 20 '19 at 16:20
@Gkm Did below answers help you fix this case, Could you share any updates? — Levi Lu-MSFT, Nov 28 '19 at 08:27
@Christine, it is just the opposite for me. Most everything runs fine at AzDo when I set the WebDriver to headed mode and the release is checked "Test mix contains UI tests". But when I specify WebDriver Headless=true, then tests that follow the press of the [Save] button fail. There is something with processing of the button that I have yet to resolve. There are other stack overflow tests that go over this phenomenon. — donvnielsen, Apr 30 '20 at 22:46
@donvnielsen Your issue in headless mode may be related to some of the settings that are enabled - I have personally seen issues with headless mode where a browser size was not specified, resulting in a small default browser size which did not render the page correctly. Feel free to post your chrome options upon driver initialization if you would like some help troubleshooting those settings. — CEH, May 06 '20 at 14:49

score 1 · Answer 1 · answered Nov 22 '19 at 17:14

I would recommend following:

When ever your tests fail take a snapshot at that moment. sometimes when you run your tests locally the network communication is a bit faster than running on server. Which means a spinner or a loading control may not appear locally because results were retrieved fast enough but the loader will appear when there is delay from server in sending the response to client.
If you are trying to run your tests in FF headless on remote grid setup then it is not yet stable in my experience.
Identify the pattern in failed scenarios which frequently fail and try and run them multiple times in a loop and identify the pattern of failure. In one of my case there was an actual bug in the performance of the application which used to get reproduced whenever the tests ran immediately in bulk

score 1 · Answer 2 · answered Mar 20 '20 at 17:52

1

I was having this issue, except my test would always fail in Azure Pipelines but pass locally. Not running in "headless" mode fixed my problem. To do this, make sure "Test mix contains UI tests" is checked in your "Visual Studio Test" task.

answered Mar 20 '20 at 17:52

georgiaboy82

554
7
13

score 0 · Answer 3 · answered Nov 22 '19 at 14:51

There are a couple of things you need to take care as follows:

Test cases fail while not finding a link or any validation messages after a button click: A lot depends on the attributes of the button. In case the button contains either of the following attributes/events the DOM Tree would be getting updated and subsequently trying to find the desired element may fail:
- onclick() event:
```
<button onclick="myFunction()">Click me</button>
```
- jQuery click() Event:
```
$("p").click(function(){
   alert("The paragraph was clicked.");
});
```
- So after a button click, to find the desired element you need to induce you have to induce WebDriverWait for the ElementExists().
- You can find a relevant discussion in ExpectedConditions.ElementIsVisible returns TimeoutException even when element is present
ExecuteScript("return document.readyState").Equals("complete"): This line of code is not that effective to ensure if the desired element(s) are present/visible/interactable with in the HTML DOM.
- You can find a relevant discussion in Do we have any generic function to check if page has completely loaded in Selenium
However to wait for specific div with ElementExists() should have worked. The relevant HTML would have helped us to analyze what went wrong. However, if the validationErrors related element is based on HTML5 Constraint validation then you have to use the GetAttribute() method.
- You can find a relevant discussion in How to handle html5 constraint validation pop-up using Selenium?

Selenium tests fail with azure pipeline sometimes and pass locally

3 Answers3