0

Note

  1. Issue started appearing only after updating TestNG version from 6.8.8 to TestNG 6.11
  2. Issue occurs even with the latest versions of TestNG 7.0.0-beta1 and maven surefire 3.0.0-M1
  3. Issue is intermittent and cannot be reproduced for all runs even with the exact same suite,tests and parameters
  4. The NPE occurs only after all the tests are completed and appears just before the @AfterSuite method is executed.
  5. The Tests consist of selenium scripts which are run on a grid.

Not really sure why the issue is not reproducible always and why apparently(based on the strack trace) the test is returning null in XmlSuite Class in TestNG. Issue occurs even without using any custom listeners

POM.xml config for surefire and testng:

<dependency>
    <groupId>org.testng</groupId>
    <artifactId>testng</artifactId>
    <version>7.0.0-beta1</version>
</dependency> 
<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>3.0.0-M1</version>
    <configuration>
        <!--<excludes> <exclude>**/integration/*Test.java</exclude> </excludes> 
            <suiteXmlFiles> <suiteXmlFile>src/test/resources/suite.xml</suiteXmlFile> 
            </suiteXmlFiles> -->
        <skip>true</skip>
        <argLine>-Xms4096m</argLine>
        <systemProperties>
            <property>
                <name>app.env</name>
                <value>${app.env}</value>
            </property>
        </systemProperties>
        <workingDirectory>./</workingDirectory>
        <reportsDirectory>${user.home}/report/reports_${report.directory}/</reportsDirectory>
    </configuration>
    <executions>
        <execution>
            <id>integration-test</id>
            <phase>integration-test</phase>
            <goals>
                <goal>test</goal>
            </goals>
            <configuration>
                <properties>
                    <property>
                        <name>usedefaultlisteners</name>
                        <value>false</value>
                    </property>
                    <property>
                        <name>listener</name>
                        <value></value>
                    </property>
                </properties>
                <skip>false</skip>
                <excludes>
                    <exclude>none</exclude>
                </excludes>
                <suiteXmlFiles>
                    <suiteXmlFile>src/test/resources/${test.suite}</suiteXmlFile>
                </suiteXmlFiles>
                <printSummary>false</printSummary> 
            </configuration>
        </execution>
    </executions>
</plugin>

Suite.XML

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Failed suite" parallel="tests" thread-count="5" verbose="3">
    <test name="Test1" preserve-order="true">
        <parameter name="country" value="us" />
        <parameter name="language" value="en" />
        <parameter name="env" value="" />
        <parameter name="browser" value="" />
        <classes>
            <class name="com.test.tests">
                <methods>
                    <include name="test1" />
                </methods>
            </class>
        </classes>
    </test>
    <test name="Test2" preserve-order="true">
        <parameter name="country" value="us" />
        <parameter name="language" value="en" />
        <parameter name="env" value="" />
        <parameter name="browser" value="" />
        <classes>
            <class name="com.test.tests">
                <methods>
                    <include name="test2" />
                </methods>
            </class>
        </classes>
    </test>
    <test name="Test3" preserve-order="true">
        <parameter name="country" value="us" />
        <parameter name="language" value="en" />
        <parameter name="env" value="" />
        <parameter name="browser" value="" />
        <classes>
            <class name="com.test.tests">
                <methods>
                    <include name="test3" />
                </methods>
            </class>
        </classes>
    </test>
    <test name="Test4" preserve-order="true">
        <parameter name="country" value="us" />
        <parameter name="language" value="en" />
        <parameter name="env" value="" />
        <parameter name="browser" value="" />
        <classes>
            <class name="com.test.tests">
                <methods>
                    <include name="test4" />
                </methods>
            </class>
        </classes>
    </test>
    <test name="Test5" preserve-order="true">
        <parameter name="country" value="us" />
        <parameter name="language" value="en" />
        <parameter name="env" value="" />
        <parameter name="browser" value="" />
        <classes>
            <class name="com.test.tests">
                <methods>
                    <include name="test5" />
                </methods>
            </class>
        </classes>
    </test>
</suite>

FULL STACK TRACE

java.lang.NullPointerException
    at org.testng.xml.XmlSuite.getAllParameters(XmlSuite.java:434)
    at org.testng.SuiteRunner.privateRun(SuiteRunner.java:367)
    at org.testng.SuiteRunner.run(SuiteRunner.java:302)
    at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:53)
    at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:96)
    at org.testng.TestNG.runSuitesSequentially(TestNG.java:1145)
    at org.testng.TestNG.runSuitesLocally(TestNG.java:1067)
    at org.testng.TestNG.runSuites(TestNG.java:997)
    at org.testng.TestNG.run(TestNG.java:965)
    at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:283)
    at org.apache.maven.surefire.testng.TestNGXmlTestSuite.execute(TestNGXmlTestSuite.java:75)
    at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:120)
    at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
    at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
    at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
    at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
ApexK
  • 1
  • 2
  • @Krishnan Mahadevan your inputs would be much appreciated since you are a testng contributor. – ApexK Dec 18 '18 at 12:35
  • I have tried reproducing this error using TestNG 7.0.0-beta1 and I cant recreate it. I would request you to please upload a simple standalone test project which we can use to recreate this issue to github.com and share that link. There's obviously something thats missing here (My hunch is that there's a listener in your test project which is messing up with things) – Krishnan Mahadevan Dec 18 '18 at 14:56
  • @KrishnanMahadevan I figured out the reason for the NPE, it seems like duplicate tests are being added to XmlTest during the execution if we use the XmlTest parameter in annotated methods. I have uploaded a small sample maven project to testng gihub issues board here : https://github.com/cbeust/testng/issues/1994 – ApexK Dec 31 '18 at 06:32
  • I have raised a [pull request](https://github.com/cbeust/testng/pull/1996) to fix this issue. – Krishnan Mahadevan Jan 01 '19 at 14:42

0 Answers0