Ensure tests continue execution when one fails in a TestNG suite

Question

I have a suite of regression tests using testng:-

<suite>
    <test>
        <classes>
            <class name="test1"/>
            <class name="test2"/>
            <class name="test3"/>
            <class name="test4"/>
            <class name="test5"/>
        </classes>
    </test>
</suite>

I have a flickering test at test 3. Sometimes it throws a NoSuchElementException (these are selenium tests). The issue arises when such an exception is encountered but not when an assertion fails - in this case the suite continues.

Sometimes, the suite continues to test 4, sometimes the whole suite stops when the failure occurs.

why is the behaviour inconsistent?
how can I force the suite to always continue to the next test when such an exception is thrown in a given class?

Example Test Class:-

public class ItineraryViewTest extends TestBase {

@BeforeClass
@Parameters({"login", "username", "password", "userReference"}) 
public void addAPackage(String login, String username, String password){

    if (Objects.equals(login, "true")) {
        loginPage().loginWith(username, password, userReference);
    }


@AfterMethod
public void closeItineraryView(){
    itineraryPage().closeItineraryView();
}

@Test(description = "Tests Itinerary view can be selected")
public void itineraryViewCanBeSelected() {
    itineraryPage().selectItineraryView();

    assertTrue("Itinerary view was not displayed",
            (itineraryPage().itineraryViewDisplayed()));
    }
}

Base class:

public class TestBase 
@BeforeClass
@Parameters({"env", "browser", "login", "mode", "emulatorMode"})
public void initialiseTests(String env, String browser, String login, String mode, String emulatorMode) throws MalformedURLException {
    APPLICATION_LOGS.debug("Running @BeforeClass");
    EnvironmentConfiguration.populate(env);
    WebDriverConfigBean webDriverConfig = aWebDriverConfig()
            .withBrowser(browser)
            .withDeploymentEnvironment(env)
            .withSeleniumMode(mode);

   //new driver gets insantiated here in this openBrowser method
    driver = WebDriverManager.openBrowser(webDriverConfig, getClass()); 

 //unrelated code...

@AfterClass
public void afterClass() {
    APPLICATION_LOGS.debug("Running @AfterClass");
    driver.quit();
}

@AfterSuite()
public void afterSuite() {
    APPLICATION_LOGS.debug("Running @AfterSuite");
    extent.flush();
}

Could you provide the stacktrace? – juherr Jan 12 '17 at 13:13 — juherr, Jan 12 '17 at 13:13

Cathal · Answer 1 · 2017-01-12T13:59:29.210

0

What version of testng are you using? there was a bug in older version(6.9.something) that caused this issue.

I'd suggest verifying you are using the latest version.

Ok, nothing is jumping out from your code or stacktrace...so it really could be any number of reasons. My only other recommendation would be to set your test steps to always run....like this:

@Test(alwaysRun = true)

edited Jan 12 '17 at 13:59

answered Jan 12 '17 at 09:56

Cathal

1,318
1
12
19

thank you, I am using 6.9.10. I will upgrade and try again – Steerpike Jan 12 '17 at 10:07
Unfortunately, I'm still getting the same problem – Steerpike Jan 12 '17 at 12:30
that sucks, can you show me how your test classes are implemented – Cathal Jan 12 '17 at 13:20
not sure whats causing the issue..check out my edit for a final suggestion – Cathal Jan 12 '17 at 13:59
thanks, would I have to add that to all @Test annotated methods in the suite or can I set it at a higher level of granularity? – Steerpike Jan 12 '17 at 14:01
Not sure off the top of my head...i do know that alwaysRun introduces a soft dependency so you are not requiring all tests to pass..check out the docs for detailed info http://testng.org/doc/documentation-main.html – Cathal Jan 12 '17 at 14:04
1

I have added alwaysrun=true to the @BeforeClass of each test class as each class has one such method. Lets's see if this forces continuation of the test run when run as a suite, following an error such as the exception I mentioned – Steerpike Jan 12 '17 at 14:11
I've realised that, following a script failing, the subsequent classes are failing with a null pointer for the driver instance. However, I create a new driver instance in the @BeforeClass annotated method in the Base class so each 'script' should get a shiny new driver. With current implementation, it appears each Class file is trying to share the same instance which fails because it is 'nulled' at the point it fails – Steerpike Jan 12 '17 at 16:44
I think I've found my answer: http://stackoverflow.com/questions/2132734/beforeclass-and-inheritance-order-of-execution – Steerpike Jan 12 '17 at 16:59

Ensure tests continue execution when one fails in a TestNG suite

1 Answers1