I am generating failsafe and surefire reports with maven with the JaCoCo plugin, but I can only manage to get them in separate reports. I would like to have a overall coverage view (merge between both Unit Tests and Integration Tests) as well.
After what I think was a thorough google search I could only found a way to do this with Sonar. Is there any simpler way to do this?
Related question: Maven separate Unit Test and Integration Tests
I recently implemented this: after some headaches and a lot of testing, I have a configuration that works beautifully.
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<version>${jacoco.version}</version>
<executions>
<execution>
<id>pre-unit-test</id>
<goals>
<goal>prepare-agent</goal>
</goals>
<configuration>
<destFile>${project.build.directory}/coverage-reports/jacoco-ut.exec</destFile>
<propertyName>surefireArgLine</propertyName>
</configuration>
</execution>
<execution>
<id>pre-integration-test</id>
<goals>
<goal>prepare-agent-integration</goal>
</goals>
<configuration>
<destFile>${project.build.directory}/coverage-reports/jacoco-it.exec</destFile>
<propertyName>testArgLine</propertyName>
</configuration>
</execution>
<execution>
<id>post-integration-test</id>
<phase>post-integration-test</phase>
<goals>
<goal>report</goal>
</goals>
<configuration>
<dataFile>${project.build.directory}/coverage-reports/jacoco-it.exec</dataFile>
<outputDirectory>${project.reporting.outputDirectory}/jacoco-it</outputDirectory>
</configuration>
</execution>
<execution>
<id>post-unit-test</id>
<phase>prepare-package</phase>
<goals>
<goal>report</goal>
</goals>
<configuration>
<dataFile>${project.build.directory}/coverage-reports/jacoco-ut.exec</dataFile>
<outputDirectory>${project.reporting.outputDirectory}/jacoco-ut</outputDirectory>
</configuration>
</execution>
<execution>
<id>merge-results</id>
<phase>verify</phase>
<goals>
<goal>merge</goal>
</goals>
<configuration>
<fileSets>
<fileSet>
<directory>${project.build.directory}/coverage-reports</directory>
<includes>
<include>*.exec</include>
</includes>
</fileSet>
</fileSets>
<destFile>${project.build.directory}/coverage-reports/aggregate.exec</destFile>
</configuration>
</execution>
<execution>
<id>post-merge-report</id>
<phase>verify</phase>
<goals>
<goal>report</goal>
</goals>
<configuration>
<dataFile>${project.build.directory}/coverage-reports/aggregate.exec</dataFile>
<outputDirectory>${project.reporting.outputDirectory}/jacoco-aggregate</outputDirectory>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.18.1</version>
<configuration>
<argLine>${surefireArgLine}</argLine>
<skipTests>${skip.unit.tests}</skipTests>
<includes>
<include>**/*UT.java</include>
<include>**/*MT.java</include>
<include>**/*Test.java</include>
</includes>
<skipTests>${skipUTMTs}</skipTests>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<version>2.12.4</version>
<configuration>
<skipTests>${skipTests}</skipTests>
<skipITs>${skipITs}</skipITs>
<argLine>${testArgLine}</argLine>
<excludes>
<exclude>**/*UT*.java</exclude>
</excludes>
</configuration>
<executions>
<execution>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
</execution>
</executions>
</plugin>
As you can see, there are 6 distinct Jacoco executions to run the tests, merge the files and create an aggregate report. On top of the Jacoco config, you also need to configure Surefire and Failsafe to take an argument from Jacoco (Surefire runs the unit tests and Failsafe runs the integration tests).
All of the configuration that I used should be there, what you do with it is your design architecture to make it fit within your desired requirements. Personally, I recommend a look into what I exclude and include within surefire and failsafe if you are having issues with files not being read.
Not the answer you're looking for, but still...
In most cases, you should not merge coverage from unit and integration tests.
The value of unit tests is that they improve the design of your application and ensure that corner cases of your code is working correctly. You should try to have a high branch coverage of your unit tests.
The value of your integration tests is that they ensure that the main use cases of your application are working correctly and that the whole stack is integrated correctly. You should try to have a high functional coverage for your integration tests. (But it is fairly hard to measure functional coverage with a tool).
If you need integration tests to improve your branch coverage, that's a strong indication that you should review the design of your code. And if you already have a high branch coverage without integration tests, adding them should not modify significantly your metrics.
You should take a look into the documentation of JaCoCo Maven plugin which contains a merge goal.
Building on top of the answer of Chad, here is my configuration. It is configured as a separate profile named jacoco so I can turn it on and off easily. It uses only the default configuration. It can create a separate code coverage report for unit tests, separate code coverage report for integration tests and also a combined code coverage report.
<profile>
<id>jacoco</id>
<build>
<plugins>
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<executions>
<execution>
<id>pre-unit-test</id>
<goals>
<goal>prepare-agent</goal>
</goals>
</execution>
<execution>
<id>post-unit-test</id>
<phase>test</phase>
<goals>
<goal>report</goal>
</goals>
</execution>
<execution>
<id>pre-integration-test</id>
<goals>
<goal>prepare-agent-integration</goal>
</goals>
</execution>
<execution>
<id>post-integration-test</id>
<goals>
<goal>report-integration</goal>
</goals>
</execution>
<execution>
<id>merge-results</id>
<phase>verify</phase>
<goals>
<goal>merge</goal>
</goals>
<configuration>
<fileSets>
<fileSet>
<directory>${project.build.directory}</directory>
<includes>
<include>*.exec</include>
</includes>
<excludes>
<exclude>aggregate.exec</exclude>
</excludes>
</fileSet>
</fileSets>
<destFile>${project.build.directory}/aggregate.exec</destFile>
</configuration>
</execution>
<execution>
<id>post-merge-report</id>
<phase>verify</phase>
<goals>
<goal>report</goal>
</goals>
<configuration>
<dataFile>${project.build.directory}/aggregate.exec</dataFile>
<outputDirectory>${project.reporting.outputDirectory}/jacoco-aggregate</outputDirectory>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
Maybe more nice way is to achieve is use same jacoco file, but let it another tests, which works for us - unit and it tests.
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<version>0.8.1</version>
<executions>
<execution>
<id>default-prepare-agent</id>
<goals>
<goal>prepare-agent</goal>
</goals>
<configuration>
<skip>${maven.surefire.skipTests}</skip>
<propertyName>maven.surefire.argLine</propertyName>
<!-- using the same dest file for both UT and IT -->
<destFile>${sonar.jacoco.reportPath}</destFile>
</configuration>
</execution>
<execution>
<id>default-prepare-agent-integration</id>
<goals>
<goal>prepare-agent-integration</goal>
</goals>
<configuration>
<skip>${maven.failsafe.skipTests}</skip>
<propertyName>maven.failsafe.argLine</propertyName>
<!-- append to the UT dest file -->
<destFile>${sonar.jacoco.reportPath}</destFile>
<append>true</append>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<skipTests>${maven.surefire.skipTests}</skipTests>
<failIfNoTests>${maven.surefire.failIfNoTests}</failIfNoTests>
<!-- allow argLine to be modified by other plugins, e.g. jacoco -->
<argLine>@{maven.surefire.argLine}</argLine>
</configuration>
</plugin>
<plugin>
<artifactId>maven-failsafe-plugin</artifactId>
<configuration>
<skipTests>${maven.failsafe.skipTests}</skipTests>
<failIfNoTests>${maven.failsafe.failIfNoTests}</failIfNoTests>
<!-- allow argLine to be modified by other plugins, e.g. jacoco -->
<argLine>@{maven.failsafe.argLine}</argLine>
</configuration>
</plugin>
This works out-of-the-box.
The prepare-agent goal integrates by default with both SureFire and FailSafe plugins (to be exact the argLine parameter set by prepare-agent is consumed by both plugins). Additionally, prepare-agent by default configures Jacoco so that it appends coverage results if the file already exists, so in the end you get single target/jacoco.exec file which contains results from both unit test and integration tests, combined.
Just in case this is the relevant config, as you see nothing changed from default:) :
<plugins>
<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.22.2</version>
<executions>
<execution>
<goals>
<goal>test</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<artifactId>maven-failsafe-plugin</artifactId>
<version>2.22.2</version>
<executions>
<execution>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<version>0.8.6</version>
<executions>
<execution>
<goals>
<goal>prepare-agent</goal>
<goal>report</goal>
<goal>check</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
I get the point where Unit tests should really be at the source of code coverage... but sometimes, one would like to have the ability to know right? Here is what I did (this is from using gradle).
plugins {
[...]
id 'jacoco'
}
[...]
test {
jacoco { /* specify the "exec" data file name (see the "Unit" in there) */
destinationFile = file("$buildDir/jacoco/jacocoTestUnit.exec")
classDumpDir = file("$buildDir/jacoco/classpathdumpsUnit")
}
useJUnitPlatform { /* I use the JUnit Jupiter @Tag Annotation to create my suites... you are free to do something else */
excludeTags 'IT'
}
description = "Run unit tests"
group = "verification"
}
task intTest(type: Test) {
jacoco {
destinationFile = file("$buildDir/jacoco/jacocoTestInt.exec")
classDumpDir = file("$buildDir/jacoco/classpathdumpsInt")
}
useJUnitPlatform {
includeTags 'IT'
}
description = "Run integration tests"
group = "verification"
}
jacocoTestReport {
/*
* find all test exec files and feed them to jacoco
*/
def list = []
def dir = new File("${buildDir}/jacoco/")
if(dir.exists()) {
dir.eachFileRecurse(FileType.FILES) { file ->
/* gather all the "exec" files available */
if (file.getName().startsWith("jacocoTest") && file.getName().endsWith(".exec")) {
list << file.getAbsolutePath()
}
}
/* provide all the "exec" files to jacoco */
executionData.from = files(list)
}
/*
* you must run all tests before running jacoco.
* We want the liberty to run partial tests instead of all,
* so this task doesn't depend on any test task.
*/
reports {
xml.enabled true
}
}
With this, you can have the the possibility to get the coverage from Unit tests with:
./gradlew clean test jacocoTestReport
Or you can have the coverage from Integration tests with:
./gradlew clean intTest jacocoTestReport
Or you can have the overall coverage of both unit and Integration tests with:
./gradlew clean test inTest jacocoTestReport
Disclaimer: I'm no Jacoco or Gradle expert... feel free to comment anything I may have forgotten. This worked beautifully so far for my needs.
In order to merge the reports here, I have a complete working solution.
Please note that in order to work properly the merge strategy, the phases should be executed sequentially (if the mvn test and mvn verify -DskipUnitTests will be executed in parallel there is possible to not work fine).
<!-- Jacoco is used to generate the reports for SonarQube -->
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<version>0.8.6</version>
<configuration>
<skip>${skipTests}</skip>
</configuration>
<executions>
<!-- Prepares the property pointing to the JaCoCo runtime agent which
is passed as VM argument when Maven the Surefire plugin is executed. -->
<execution>
<id>pre-unit-test</id>
<goals>
<goal>prepare-agent</goal>
</goals>
<configuration>
<!-- Sets the path to the file which contains the execution data. -->
<destFile>${project.build.directory}/coverage-reports/jacoco-ut.exec</destFile>
<!-- Sets the name of the property containing the settings for JaCoCo runtime agent. -->
<propertyName>surefireArgLine</propertyName>
</configuration>
</execution>
<!-- Make sure that after the Unit Tests execution, the jacoco-ut.exec file is generated,
will be merged to the aggregation file -->
<execution>
<id>post-unit-merge</id>
<phase>test</phase>
<goals>
<goal>merge</goal>
</goals>
<configuration>
<fileSets>
<fileSet>
<directory>${project.build.directory}</directory>
<includes>
<include>**/coverage-reports/*.exec</include>
</includes>
</fileSet>
</fileSets>
<destFile>${project.build.directory}/coverage-reports/jacoco_aggregation.exec</destFile>
</configuration>
</execution>
<!-- Ensures that the code coverage report is created/updated after unit tests have been run. -->
<execution>
<id>post-unit-test</id>
<phase>test</phase>
<goals>
<goal>report</goal>
</goals>
<configuration>
<!-- Sets the path to the file which contains the execution data. -->
<dataFile>${project.build.directory}/coverage-reports/jacoco_aggregation.exec</dataFile>
<!-- Sets the output directory for the code coverage report. -->
<outputDirectory>${project.reporting.outputDirectory}/jacoco</outputDirectory>
</configuration>
</execution>
<!-- Prepares the property pointing to the JaCoCo runtime agent which
is passed as VM argument when Maven the Failsafe plugin is executed. -->
<execution>
<id>pre-integration-test</id>
<phase>pre-integration-test</phase>
<goals>
<goal>prepare-agent</goal>
</goals>
<configuration>
<!-- Sets the path to the file which contains the execution data. -->
<destFile>${project.build.directory}/coverage-reports/jacoco-it.exec</destFile>
<!-- Sets the name of the property containing the settings for JaCoCo runtime agent. -->
<propertyName>failsafeArgLine</propertyName>
</configuration>
</execution>
<!-- Make sure that after the Integration Test execution, the jacoco-it.exec file is generated,
will be merged to the aggregation file -->
<execution>
<id>post-integration-merge</id>
<phase>post-integration-test</phase>
<goals>
<goal>merge</goal>
</goals>
<configuration>
<fileSets>
<fileSet>
<directory>${project.build.directory}</directory>
<includes>
<include>**/coverage-reports/*.exec</include>
</includes>
</fileSet>
</fileSets>
<destFile>${project.build.directory}/coverage-reports/jacoco_aggregation.exec</destFile>
</configuration>
</execution>
<!-- Ensures that the code coverage report is created/updated after integration tests have been run. -->
<execution>
<id>post-integration-test</id>
<phase>post-integration-test</phase>
<goals>
<goal>report</goal>
</goals>
<configuration>
<!-- Sets the path to the file which contains the execution data. -->
<dataFile>${project.build.directory}/coverage-reports/jacoco_aggregation.exec</dataFile>
<!-- Sets the output directory for the code coverage report. -->
<outputDirectory>${project.reporting.outputDirectory}/jacoco</outputDirectory>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<!-- Sets the VM argument line used when unit tests are run. -->
<argLine>${surefireArgLine}</argLine>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<configuration>
<!-- Sets the VM argument line used when integration tests are run. -->
<argLine>${failsafeArgLine}</argLine>
</configuration>
</plugin>
Now, since the reports are generated, can be executed the sonar command in order to publis the report:
mvn sonar:sonar -Dsonar.coverage.jacoco.xmlReportPaths="target/site/jacoco/jacoco.xml"
Below configuration works well and generates an aggregated code coverage report comprising code covered by unit and integration tests.
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<version>0.8.10</version>
<executions>
<execution>
<id>pre-unit-test</id>
<goals>
<goal>prepare-agent</goal>
</goals>
<configuration>
<append>true</append>
<destFile>${project.build.directory}/jacoco-ut.exec</destFile>
</configuration>
</execution>
<execution>
<id>post-unit-test</id>
<phase>test</phase>
<goals>
<goal>report</goal>
</goals>
<configuration>
<dataFile>${project.build.directory}/jacoco-ut.exec</dataFile>
<outputDirectory>${project.reporting.outputDirectory}/jacoco-ut</outputDirectory>
</configuration>
</execution>
<execution>
<id>pre-integration-test</id>
<goals>
<goal>prepare-agent-integration</goal>
</goals>
<configuration>
<append>true</append>
<destFile>${project.build.directory}/jacoco-it.exec</destFile>
</configuration>
</execution>
<execution>
<id>post-integration-test</id>
<phase>post-integration-test</phase>
<goals>
<goal>report</goal>
</goals>
<configuration>
<dataFile>${project.build.directory}/jacoco-it.exec</dataFile>
<outputDirectory>${project.reporting.outputDirectory}/jacoco-it</outputDirectory>
</configuration>
</execution>
<execution>
<id>merge-all-results</id>
<phase>verify</phase>
<goals>
<goal>merge</goal>
</goals>
<configuration>
<fileSets>
<fileSet>
<directory>${project.build.directory}</directory>
<includes>
<include>jacoco*.exec</include>
</includes>
</fileSet>
</fileSets>
<destFile>${project.build.directory}/jacoco-aggregate.exec</destFile>
</configuration>
</execution>
<execution>
<id>merged-report</id>
<phase>verify</phase>
<goals>
<goal>report</goal>
</goals>
<configuration>
<dataFile>${project.build.directory}/jacoco-aggregate.exec</dataFile>
<outputDirectory>${project.reporting.outputDirectory}/jacoco-aggregate</outputDirectory>
</configuration>
</execution>
<execution>
<id>code-coverage-report-check</id>
<goals>
<goal>check</goal>
</goals>
<configuration>
<dataFile>${project.build.directory}/jacoco-aggregate.exec</dataFile>
<rules>
<rule>
<element>BUNDLE</element>
<limits>
<limit>
<counter>COMPLEXITY</counter>
<value>COVEREDRATIO</value>
<minimum>0.9</minimum>
</limit>
</limits>
</rule>
</rules>
</configuration>
</execution>
</executions>
</plugin>
all answers above a way too complicated an totaly unnecessary, because:
The only thing you have to make sure is, that the report is generated after the integration-test phase. Because by default the Jacoco report is run before the integration-test phase.
Surprisingly enough, this:
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<version>0.8.10</version>
<executions>
<execution>
<id>jacoco-prepare-agent</id>
<goals>
<goal>prepare-agent</goal>
</goals>
</execution>
<execution>
<id>jacoco-report</id>
<!-- needed in order to make the report also include the results from the integration tests -->
<phase>post-integration-test</phase>
<goals>
<goal>report</goal>
</goals>
</execution>
</executions>
</plugin>
already adds the additional necessary command line parameters for coverage measuring not only to the unit-tests but also to the integration-tests. But since the report is generated by default before integration-tests their coverage is not included in the test by default.
So just adding this:
<execution>
<id>jacoco-report</id>
<!-- needed in order to make the report also include the results from the integration tests -->
<phase>post-integration-test</phase>
<goals>
<goal>report</goal>
</goals>
</execution>
is already sufficient to add the integration-tests coverage to the report.
All the claims on the internet, and in chatGPT for a much more complicated configuration using Jacocos merge goal or defining <argLine>${argLine}</argLine> for surefire-plugin or <argLine>${failsafe.argLine}</argLine> for the fail-safe-plugin is not necessary at all!
