ID collision when using testrail as reporter for testng tests

Question

When posting test results from testgn to test rail, I am running into an issue with how to manage unique test case ids.

I had initially stored them in the test method, so each method mapped to a testrail test case. This worked until I started parameterizing the test methods.

Now let's say I want to run a single selenium test on four different browsers, and store the results separately. I cannot store the case id in the test method, as the four different results would report back to the same test case.

Instead, I decided to try adding the case ID as a parameter in the .xml file. This works, but only as long as you have a single test method per class, otherwise testXXX() and testYYY() get the same id for each entry in the xml. So no luck there either.

I'm trying to find a way to store case ids for every version of every test run, without making structural sacrifices (such as abandoning parameters, or writing only a single test method per class).

A sample suite is below

<suite name="UL" parallel="tests" thread-count="1" verbose="10">
    <parameter name="env" value="REDACTED"/>
    <parameter name="recordTests" value="1"/>
    <listeners>
    </listeners>
    <test name="UL Tests firefox">
        <classes>
            <class name="tests.selenium_tests.ULTests">
                <parameter name="browser" value="firefox"/>
                <parameter name="case_id" value="1111"/>
            </class>
        </classes>
    </test>
    <test name="UL Tests chrome">
        <classes>
            <class name="tests.selenium_tests.ULTests">
                <parameter name="browser" value="chrome"/>
                <parameter name="case_id" value="1112"/>
            </class>
        </classes>
    </test>
    <test name="UL Tests safari">
        <classes>
            <class name="tests.selenium_tests.ULTests">
                <parameter name="browser" value="bs_safari"/>
            </class>
        </classes>
    </test>
    <test name="UL Tests edge">
        <classes>
            <class name="tests.selenium_tests.ULTests">
                <parameter name="browser" value="bs_edge"/>
            </class>
        </classes>
    </test>
</suite>

Krishnan Mahadevan · Accepted Answer · 2019-01-17T17:35:26.233

It all depends upon how you visualise your TestCase ID in the TCMS system.

If a test case represents a data driven test, then the approach needs to be a bit more different.

If a test case represents a regular test, then I believe you already have a working solution.

Here's one way of getting this done. I am using TestNG 7.0.0-beta3 (latest released version as of today)

Assumption:

A Test case in TCMS represents "n" iterations of an actual test and is considered to be a pass if and only if all the iterations pass, else its a fail.

Steps to be followed:

You first create a custom annotation which captures the TCMS (Testcase Management System) ID of a particular test.
You annotate your @Test methods using the custom annotation to tie it down to a particular TCMS testcase.
You now build a custom listener which ensures that it is able to differentiate between ordinary tests and data driven tests and post the results accordingly. For data driven tests, they would need to keep track of all the iterations that have run so far and then compute the overall results.

The same is elaborated in my blog post here.

Here's a sample that shows all of this in action:

The custom annotation looks like this:

import static java.lang.annotation.ElementType.METHOD;
import static java.lang.annotation.ElementType.TYPE;

import java.lang.annotation.Retention;
import java.lang.annotation.Target;

@Retention(java.lang.annotation.RetentionPolicy.RUNTIME)
@Target({METHOD, TYPE})
public @interface Tcms {
  String id() default "";
}

The listener looks like below:

import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
import org.testng.IInvokedMethod;
import org.testng.IInvokedMethodListener;
import org.testng.ITestResult;

public class TestRailReporter implements IInvokedMethodListener {
  private Map<String, Boolean> resultTracker = new ConcurrentHashMap<>();

  @Override
  public void beforeInvocation(IInvokedMethod method, ITestResult testResult) {
    String key = testResult.getInstanceName() + "." + method.getTestMethod().getMethodName();
    resultTracker.putIfAbsent(key, Boolean.TRUE);
  }

  @Override
  public void afterInvocation(IInvokedMethod method, ITestResult testResult) {
    Tcms tcms =
        method.getTestMethod().getConstructorOrMethod().getMethod().getAnnotation(Tcms.class);
    // Only report those tests to TestRail wherein our annotation is found.
    if (tcms == null) {
      return;
    }
    if (method.getTestMethod().isDataDriven()) {
      // For data driven tests we need a different logic
      String key = testResult.getInstanceName() + "." + method.getTestMethod().getMethodName();
      if (method.getTestMethod().hasMoreInvocation()) {
        Boolean result = resultTracker.get(key);
        result = result && (testResult.getStatus() == ITestResult.SUCCESS);
        resultTracker.put(key, result);
        return;
      }
      postResultsToTestRail(tcms, resultTracker.get(key));
    } else {
      postResultsToTestRail(tcms, testResult.getStatus() == ITestResult.SUCCESS);
    }
  }

  private void postResultsToTestRail(Tcms tcms, boolean pass) {
    String testCaseId = tcms.id();
    // Write logic here that takes care of posting results to the TCMS system
    System.err.println("Test case Id [" + testCaseId + "] passed ? " + pass);
  }
}

A sample test case :

import org.testng.Assert;
import org.testng.annotations.DataProvider;
import org.testng.annotations.Listeners;
import org.testng.annotations.Test;

@Listeners(TestRailReporter.class)
public class SampleTestCase {

  @Test
  @Tcms(id = "TESTRAIL-1")
  public void testMethod() {
    Assert.assertTrue(true);
  }

  @Test(dataProvider = "dp")
  @Tcms(id = "TESTRAIL-2")
  public void dataDrivenTestWithSomeFailures(int i) {
    if (i % 2 == 0) {
      Assert.fail("simulating a failure");
    }
  }

  @Test(dataProvider = "dp")
  @Tcms(id = "TESTRAIL-3")
  public void dataDrivenTestWithNoFailures(int i) {
    Assert.assertTrue(i >= 0);
  }

  @DataProvider(name = "dp")
  public Object[][] getData() {
    return new Object[][] {{1}, {2}, {3}};
  }
}

The output:

Test case Id [TESTRAIL-3] passed ? true
Test case Id [TESTRAIL-2] passed ? false


java.lang.AssertionError: simulating a failure

    at org.testng.Assert.fail(Assert.java:97)
    at com.rationaleemotions.stackoverflow.qn54224337.SampleTestCase.dataDrivenTestWithSomeFailures(SampleTestCase.java:21)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:131)
    at org.testng.internal.TestInvoker.invokeMethod(TestInvoker.java:570)
    at org.testng.internal.TestInvoker.invokeTestMethod(TestInvoker.java:170)
    at org.testng.internal.MethodRunner.runInSequence(MethodRunner.java:46)
    at org.testng.internal.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:790)
    at org.testng.internal.TestInvoker.invokeTestMethods(TestInvoker.java:143)
    at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:146)
    at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:128)
    at org.testng.TestRunner.privateRun(TestRunner.java:763)
    at org.testng.TestRunner.run(TestRunner.java:594)
    at org.testng.SuiteRunner.runTest(SuiteRunner.java:398)
    at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:392)
    at org.testng.SuiteRunner.privateRun(SuiteRunner.java:355)
    at org.testng.SuiteRunner.run(SuiteRunner.java:304)
    at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:53)
    at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:96)
    at org.testng.TestNG.runSuitesSequentially(TestNG.java:1146)
    at org.testng.TestNG.runSuitesLocally(TestNG.java:1067)
    at org.testng.TestNG.runSuites(TestNG.java:997)
    at org.testng.TestNG.run(TestNG.java:965)
    at org.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:73)
    at org.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:123)

Test case Id [TESTRAIL-1] passed ? true

===============================================
Default Suite
Total tests run: 7, Passes: 6, Failures: 1, Skips: 0
===============================================

Edit: Based on the comments from OP here's the other way of doing this.

Approach 2

The annotation being used:

import static java.lang.annotation.ElementType.METHOD;

import java.lang.annotation.Retention;
import java.lang.annotation.Target;

@Retention(java.lang.annotation.RetentionPolicy.RUNTIME)
@Target({METHOD})
public @interface Tcms {
  String id() default "";
}

The argument that would be passed on to the test method by a data provider, for data driven tests would look like below:

import java.lang.annotation.Annotation;

public class TestData implements Tcms {

  private String tcmsId;
  private String data;

  public TestData(String tcmsId, String data) {
    this.tcmsId = tcmsId;
    this.data = data;
  }

  @Override
  public String id() {
    return tcmsId;
  }

  public String getData() {
    return data;
  }

  @Override
  public Class<? extends Annotation> annotationType() {
    return Tcms.class;
  }

  @Override
  public String toString() {
    return getData();
  }
}

The listener looks like below:

import org.testng.IInvokedMethod;
import org.testng.IInvokedMethodListener;
import org.testng.ITestResult;

public class TestRailReporter2 implements IInvokedMethodListener {

  @Override
  public void afterInvocation(IInvokedMethod method, ITestResult testResult) {
    if (method.getTestMethod().isDataDriven()) {
      //Data driven tests need to be handled differently
      Object[] parameters = testResult.getParameters();
      if (parameters.length != 1) {
        //If theres more than one parameter, then dont do anything.
        return;
      }
      Object parameter = parameters[0];
      if (!(parameter instanceof Tcms)) {
        //If the parameter doesnt implement our interface dont do anything
        return;
      }
      postResultsToTestRail(
          (Tcms) parameter, testResult.getStatus() == ITestResult.SUCCESS, parameter.toString());
    } else {
      Tcms tcms =
          method.getTestMethod().getConstructorOrMethod().getMethod().getAnnotation(Tcms.class);
      if (tcms == null) {
        return;
      }
      postResultsToTestRail(tcms, testResult.getStatus() == ITestResult.SUCCESS);
    }
  }

  private void postResultsToTestRail(Tcms tcms, boolean pass) {
    String testCaseId = tcms.id();
    // Write logic here that takes care of posting results to the TCMS system
    System.err.println("Test case Id [" + testCaseId + "] passed ? " + pass);
  }

  private void postResultsToTestRail(Tcms tcms, boolean pass, String param) {
    String id = tcms.id();
    // Write logic here that takes care of posting results to the TCMS system
    System.err.println("Test case Id [" + id + "] with parameter [" + param + "] passed ? " + pass);
  }
}

The test class looks like below:

import org.testng.Assert;
import org.testng.annotations.DataProvider;
import org.testng.annotations.Listeners;
import org.testng.annotations.Test;

@Listeners(TestRailReporter2.class)
public class AnotherSampleTestCase {

  @Test
  @Tcms(id = "TESTRAIL-1")
  public void simpleTestMethod() {
    Assert.assertTrue(true);
  }

  @Test(dataProvider = "dp")
  public void dataDrivenTestMethod(TestData data) {
    Assert.assertFalse(data.getData().trim().isEmpty());
  }

  @DataProvider(name = "dp")
  public Object[][] getData() {
    return new Object[][] {
      {new TestData("TESTRAIL-2", "Jack")},
      {new TestData("TESTRAIL-3", "")},
      {new TestData("TESTRAIL-4", "Daniels")}
    };
  }
}

Here's the execution output:

Test case Id [TESTRAIL-2] with parameter [Jack] passed ? true
Test case Id [TESTRAIL-3] with parameter [] passed ? false

java.lang.AssertionError: did not expect to find [false] but found [true]

    at org.testng.Assert.fail(Assert.java:97)
    at org.testng.Assert.failNotEquals(Assert.java:969)
    at org.testng.Assert.assertFalse(Assert.java:65)
    at org.testng.Assert.assertFalse(Assert.java:75)
    at com.rationaleemotions.stackoverflow.qn54224337.AnotherSampleTestCase.dataDrivenTestMethod(AnotherSampleTestCase.java:19)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:131)
    at org.testng.internal.TestInvoker.invokeMethod(TestInvoker.java:570)
    at org.testng.internal.TestInvoker.invokeTestMethod(TestInvoker.java:170)
    at org.testng.internal.MethodRunner.runInSequence(MethodRunner.java:46)
    at org.testng.internal.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:790)
    at org.testng.internal.TestInvoker.invokeTestMethods(TestInvoker.java:143)
    at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:146)
    at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:128)
    at org.testng.TestRunner.privateRun(TestRunner.java:763)
    at org.testng.TestRunner.run(TestRunner.java:594)
    at org.testng.SuiteRunner.runTest(SuiteRunner.java:398)
    at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:392)
    at org.testng.SuiteRunner.privateRun(SuiteRunner.java:355)
    at org.testng.SuiteRunner.run(SuiteRunner.java:304)
    at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:53)
    at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:96)
    at org.testng.TestNG.runSuitesSequentially(TestNG.java:1146)
    at org.testng.TestNG.runSuitesLocally(TestNG.java:1067)
    at org.testng.TestNG.runSuites(TestNG.java:997)
    at org.testng.TestNG.run(TestNG.java:965)
    at org.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:73)
    at org.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:123)

Test case Id [TESTRAIL-4] with parameter [Daniels] passed ? true
Test case Id [TESTRAIL-1] passed ? true

===============================================
Default Suite
Total tests run: 4, Passes: 3, Failures: 1, Skips: 0
===============================================


Process finished with exit code 0

Hi Krishnan, thanks for the detailed reply. Unfortunately the custom annotation leaves me with the same issue--we're not capturing different browser behavior in different ids. I'm looking into a custom listener at the moment that might work. As a stopgap I've built a switch into the relevant test cases, but this current solution will scale poorly. — Driver, Jan 17 '19 at 14:04
@Driver - I have updated my answer based on your comment. Please take a look now. — Krishnan Mahadevan, Jan 17 '19 at 17:35
Thanks Krishnan, did some additional tinkering, and proper situational use of data providers is working for us. — Driver, Jan 24 '19 at 14:50

ID collision when using testrail as reporter for testng tests

1 Answers1