6

Our company is developing a framework with Selenium, POM, Maven and Java for a Web application and we have around 35 test cases. When we run testng.xml there are at least 4 to 5 test cases that fail randomly because of stale element exception or element not clickable at that point, etc.

Is it common for some test cases to fail when we run testng.xml? How many test cases are ran in your organization and what is the estimate on how many that fail?

Krishnan Mahadevan
  • 14,121
  • 6
  • 34
  • 66
BretA
  • 75
  • 8
  • If test fails it means either your code under test works not how you expect or your test is bad. There should not be failed tests at all – Ivan Jan 09 '19 at 04:48

5 Answers5

1

Failures due to stale elements, element not clickable at a point, timing issues, etc. must be handled and dealt with in your automation framework - the methods you're creating and using to construct the cases.
They should not propagate and lead to case failures - they are tech issues, not a product problem, or test case one. As such they must be accounted for (try/catch blocks for example) and dealt with (retry mechanisms, or re-getting a web element) promptly.

In essence - treat these kinds of failures the same way as you treat syntax errorrs - there should not be such.

In the same time - and speaking simply out of my experience - cases dealing with live/dynamic data may sometimes randomly fail.

For instance, a SUT I'm working on displays some metrics and aggregations based on data and actions outside of my control (life traffic from upstream systems). There are cases checking a particular generated artifact behaves according to the set expecations (imagine a monthy graph for instance, which simply doesn't have a number of data points - there just was no activity on those days) - cases for it will fail, not because they were constructed incorrectly, and certainly not because there is a product bug - but because of the combination of the time of execution and the dataset.

Over time I've come to the conclusion having those failures is OK, getting them "fixed"- reselecting data sets, working around such outside fluctuations, etc. is an activity with diminishing value, and questionable ROI. Out of the current ~10,000 cases for this system, around 1.5% are failing because of this (disclaimer: the SUT is working exclusively with live/production data).
This is hardly a rule of thumb - it's just a number I've settled on as acceptable given the context.

And important note - if I had full control of this very data, I would have gotten rid of those "random" failures also. I've chosen to use the real data deliberately - thus my cases also verifying its integrity; with this negative side effect.

Todor Minakov
  • 19,097
  • 3
  • 55
  • 60
1

You just need to add some wait before your driver.findElement(). Selenium works very fast and that's why you are getting this Stale element or element not visible exceptions. Adding wait should solve the problem.

Dish
  • 117
  • 8
1

Test Automation is related to the repeatability of the tests and the speed at which the tests can be executed. There are a number of commercial and open source tools available for assisting with the development of Test Automation and Selenium is possibly the most widely-used open source solution among them.

Acceptable range of failed tests

This matrix may vary from organization to organization or as per specific Client Requirements. However Exit Criteria must hold the key for this limit. Having said that, as Test Automation through Selenium automates the Regression Tests so ideally there should be Zero failures. I have known some organization adhering to Zero Defect policy.

Errors you are facing

Conclusion

So the errors which you mentioned are not errors as such but may arise while Test Execution due to the following reasons:

  • Mismatch in the compatibility between the version of binaries you are using.
  • Synchronization mismatch between the WebDriver instance and Web Browser instance.

These error can be addressed easily following the best practices mentioned above.

undetected Selenium
  • 183,867
  • 41
  • 278
  • 352
0

Stale Element Exceptions - This exception can show up when the element is no longer in the DOM. The major contributor to this issue is the page getting loaded up when the element is being looked out for. The best way to handle this is to catch the exception. As suggested earlier, it should be designed as part of the framework to handle these cases

Element Not clickable at Point - This issue can happen due to few reasons

  1. Page is still loading
  2. There is a overlay on top of the element which we want to click , thus obstructing it

Best Approach The best way to resolve flakiness in tests is to capture the screenshots at the instance of failure and work on it. A testing framework should be designed to handle all these edge case scenarios

Anant Krish
  • 17
  • 3
0

Thanks Guys, I could address the issue

Steps followed 1) Stale element exception: Scenario: Based on a search criteria the table of contents would load with edit and delete button. The search was slow and the elements to edit button would go stale Solution : used custom wait sample code below from earlier posts and pausing it worked

public static void customewait(int seconds)
    {        Date start = new Date();
             Date end = new Date();
             while(end.getTime() - start.getTime() < seconds * 1000){
                 end = new Date();
             }
    }

Earlier used below code but did not work 

public void getVisibilityAllTableAddressBook() 
{
        WebDriverWait wait = new WebDriverWait(driver, 10);
        wait.until(ExpectedConditions.visibilityOfAllElements(driver.findElements(By.cssSelector("#data-table-items"))));

}
BretA
  • 75
  • 8