1

In order to integrate with an external test management solution and have visibility of karate results, I need to import them using the JUnit XML report. Test cases identified in JUnit's XML report are identified by the name and classname attributes on the element. However, it seems that these names are being generated somehow dynamicaly and not as unique identifiers that clearly identify the original testcase. The ideal case would be to have unique identifiers for the corresponding testcases, so if we import results multiple times they are always associated with the same entities in the test management solution.

I've tried running the karate-junit5 sample code and ran it using "mvn test". This is a sample of the generated JUnit XML report.

  <testcase name="testTags[1][1]" classname="karate.SampleTest" time="0.002"/>
  <testcase name="testSample[1][1]" classname="karate.SampleTest" time="0"/>
  <testcase name="testSample[1][2]" classname="karate.SampleTest" time="0"/>
  <testcase name="testFullPath[1][1]" classname="karate.SampleTest" time="0"/>

I'd expect that the "name" attribute would be filled with an unique identifier for the test case, independent on how many times it will be run, for example with the value of Scenario (e.g. "first hello world")

Feature: sample

  Scenario: first hello world
    * print 'hello'

  Scenario: second scenario
    * print 'second'

Thus, having a generate JUnit XML report like this would be something more useful.

  <testcase name="second" classname="karate.SampleTest" time="0.002"/>
  <testcase name="first hello world" classname="karate.SampleTest" time="0"/>
  <testcase name="second scenario" classname="karate.SampleTest" time="0"/>
  <testcase name=first" classname="karate.SampleTest" time="0"/>

Or having the name attribute being filled with the feature name plus the scenario name. Anyway, is there any way of configuring how these attributes on the element of the Junit report should be filled?

Sérgio
  • 1,777
  • 2
  • 10
  • 12

1 Answers1

0

Pretty sure you are using an old version of Karate or something else (maybe JUnit itself) generated this XML. Read this, and you may need to point your reporting solution to a different folder or exclude some files not created by Karate: https://github.com/intuit/karate#test-reports

There certainly may be opportunities to improve Karate so I propose you do this.

Open a new issue, and follow this process: https://github.com/intuit/karate/wiki/How-to-Submit-an-Issue - and then specify what changes you want to what is generated today.

Also the XML creation code is in this file Engine.java look for the saveResultXml() method.

Peter Thomas
  • 54,465
  • 21
  • 84
  • 248
  • Hi, I've used the latest version from upstream. I went to the karate-junit5 folder and ran maven using "mvn test", which generated a JUnit XML report named "target/surefire-reports/TEST-karate.SampleTest.xml" with the contents I've shared. – Sérgio Aug 06 '19 at 15:58
  • @user2850180 and I believe you :) you know what to do now, see above – Peter Thomas Aug 06 '19 at 15:59
  • was this report generated by surefire itself or by the code that you kindly shared earlier? if the later, then I guess the PR is the solution right? – Sérgio Aug 06 '19 at 16:00
  • @user2850180 I don't know - that is why you first try this locally in a fresh environment and then consider a PR. – Peter Thomas Aug 06 '19 at 16:01
  • The JUnit XML report I was looking at was the one being generated by maven surefire itself; it's not the one created by Karate. From what I saw in code, JUnit XML reports are only generated by Karate if we use the parallel runner, right? Thus if I run the tests mentioned in this class from karate-junit5 folder https://github.com/intuit/karate/blob/master/karate-junit5/src/test/java/karate/SampleTest.java ..shouldn't it call the parallel runner? It seems that it isn't calling it unless I add "Runner.parallel" – Sérgio Aug 06 '19 at 17:29
  • @user2850180 the parallel runner is different, read the docs carefully please: https://github.com/intuit/karate#junit-5-parallel-execution – Peter Thomas Aug 06 '19 at 19:10
  • thanks but I've already read them but it's still not clear to me. Are Karate built JUnit XML reports only generated if we use the parallel runner? – Sérgio Aug 07 '19 at 08:21
  • @user2850180 I have nothing more to add here. if you can't create a quick-start, maybe you should look for some other framework or try ask someone next to you to help you. all the best ! – Peter Thomas Aug 07 '19 at 13:22
  • @user2850180 see the last comment here - if you are wondering: https://stackoverflow.com/a/57059930/143475 – Peter Thomas Aug 07 '19 at 13:23