Changed JaCoCo from offline to online, but still don't see coverage for classes in other modules

Question

I have a largish Maven multi-module project build. I'm starting with all of our tests using PowerMock, and using Jacoco 0.7.8 and offline instrumentation.

Despite the fact that these are called "unit tests", the tests for classes in one module do call significant code in other modules. As a result, when we view the generated reports, we see the coverage for the classes in the same module as the test, but we don't see coverage for classes in other modules that were executed during the test.

My ASSUMPTION is that if I'm able to change this to use online instrumentation, the resulting coverage report would include the classes from other modules outside of the current module.

So, I set about fixing the one detail in our CUTs (class under test) that required the use of PowerMock instead of Mockito (that part was pretty simple). I fixed this one by one in each test and corresponding CUT, but for now I've only made these changes in a single module in the larger build. I verified that the tests work, and that I can see interactive coverage with EclEmma (another limitation of PowerMock).

The jacoco-maven-plugin is configured in the parent pom that is used by all of the child modules, to presently use offline instrumentation. Eventually, I'm going to change the config in this parent pom to use online instrumentation, but I've decided to leave that there for now and override the config in each module that I've done the conversion in, so that that module uses online instrumentation. I believe I've done that "properly", although I had to use a hack to override the executions list in the child module (remember I'm going to be deleting these once all of them are converted).

When I run the build ("clean install"), I see it do "prepare-agent", printing the resulting "argLine" value, then later on surefire executes, then the "report" goal executes, printing the path to the jacoco.exec file, and I see that it says "Analyzed bundle 'my-module-name' with 1 classes". That seems indicative of a problem.

As the build was now complete, I then opened the "target/site/jacoco/index.html" file in my browser, and I found that it only included coverage for the present module, and only the single class.

What was also curious is that I superficially inspected the generated "jacoco.exec" file. I noticed that it was definitely larger than I was used to seeing from other modules still using offline instrumentation. I don't have any way to interpret the format of the file, but I did try to simply "cat" the file, just to see what I can see. I saw strings that represented class names in other modules that my tests are executing.

So, it seems like online instrumentation is at least recording data for classes in other modules, but the resulting report doesn't show that.

Is it possible to get this coverage?

The following is an excerpt from the "effective-pom" output:

  <plugin>
    <groupId>org.jacoco</groupId>
    <artifactId>jacoco-maven-plugin</artifactId>
    <version>0.7.8</version>
    <executions>
      <execution>
        <id>default-instrument</id>
        <phase>none</phase>
        <goals>
          <goal>instrument</goal>
        </goals>
      </execution>
      <execution>
        <id>default-restore-instrumented-classes</id>
        <phase>none</phase>
        <goals>
          <goal>restore-instrumented-classes</goal>
        </goals>
      </execution>
      <execution>
        <id>default-report</id>
        <phase>prepare-package</phase>
        <goals>
          <goal>report</goal>
        </goals>
      </execution>
      <execution>
        <id>prepare-agent</id>
        <goals>
          <goal>prepare-agent</goal>
        </goals>
      </execution>
    </executions>
  </plugin>
  <plugin>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>2.19.1</version>
    <executions>
      <execution>
        <id>default-test</id>
        <phase>test</phase>
        <goals>
          <goal>test</goal>
        </goals>
        <configuration>
          <argLine>@{argLine} -Xmx1024m</argLine>
          <includes>
            <include>**/*Test.java</include>
          </includes>
          <systemPropertyVariables>
            <running-unit-test>true</running-unit-test>
            <jacoco-agent.destfile>...\target/jacoco.exec</jacoco-agent.destfile>
          </systemPropertyVariables>
        </configuration>
      </execution>
    </executions>
    <configuration>
      <argLine>@{argLine} -Xmx1024m</argLine>
      <includes>
        <include>**/*Test.java</include>
      </includes>
      <systemPropertyVariables>
        <running-unit-test>true</running-unit-test>
        <jacoco-agent.destfile>...\target/jacoco.exec</jacoco-agent.destfile>
      </systemPropertyVariables>
    </configuration>
  </plugin>

Notice that the executions used for offline instrumentation have phase "none", so they aren't used, and I'm also setting the "jacoco-agent.destfile" property, which is only used for offline instrumentation.

Answer 1

Goal report performs analysis of classes only in current module and creates report containing only them, even if data was recorded for other classes. And goal report-aggregate

coverage report from multiple projects within reactor

(see below for example)

Also coming back to PowerMockito - as was said in https://stackoverflow.com/a/42305077/244993 it adds additional complexity, however:

a/src/main/java/example/A.java:

package example;

class A {
  // to be tested in module "a"
  void a() {
    System.out.println("A_a" + fun());
  }

  // to be tested in module "b"
  void b() {
    System.out.println("A_b" + fun());
  }

  // to be mocked by PowerMockito
  static String fun() {
    return "";
  }
}

a/src/test/java/example/ATest.java:

package example;

import org.junit.Test;
import org.junit.runner.RunWith;
import org.mockito.Mockito;
import org.powermock.api.mockito.PowerMockito;
import org.powermock.core.classloader.annotations.PrepareForTest;
import org.powermock.modules.junit4.PowerMockRunner;

@PrepareForTest({A.class})
@RunWith(PowerMockRunner.class)
public class ATest {
  @Test
  public void test() {
    PowerMockito.mockStatic(A.class);
    Mockito.when(A.fun()).thenReturn("fun");

    new A().a();
  }
}

b/src/main/java/example/B.java:

package example;

class B {
  // to be tested in module "b"
  void b() {
    System.out.println("B_b" + fun());
  }

  // to be mocked by PowerMockito
  static String fun() {
    return "";
  }
}

b/src/test/java/example/BTest.java:

package example;

import org.junit.Test;
import org.junit.runner.RunWith;
import org.mockito.Mockito;
import org.powermock.api.mockito.PowerMockito;
import org.powermock.core.classloader.annotations.PrepareForTest;
import org.powermock.modules.junit4.PowerMockRunner;

@PrepareForTest({A.class, B.class})
@RunWith(PowerMockRunner.class)
public class BTest {
  @Test
  public void test() {
    PowerMockito.mockStatic(A.class);
    Mockito.when(A.fun()).thenReturn("fun");

    PowerMockito.mockStatic(B.class);
    Mockito.when(B.fun()).thenReturn("fun");

    new A().b();
    new B().b();
  }
}

a/pom.xml:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <parent>
    <groupId>org.example</groupId>
    <artifactId>example</artifactId>
    <version>1.0-SNAPSHOT</version>
  </parent>

  <artifactId>a</artifactId>
</project>

b/pom.xml:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <parent>
    <groupId>org.example</groupId>
    <artifactId>example</artifactId>
    <version>1.0-SNAPSHOT</version>
  </parent>

  <dependencies>
    <dependency>
      <groupId>org.example</groupId>
      <artifactId>a</artifactId>
      <version>${project.version}</version>
    </dependency>
  </dependencies>

  <artifactId>b</artifactId>
</project>

report/pom.xml for aggregation of coverage across modules:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <parent>
    <groupId>org.example</groupId>
    <artifactId>example</artifactId>
    <version>1.0-SNAPSHOT</version>
  </parent>

  <artifactId>report</artifactId>
  <packaging>pom</packaging>

  <dependencies>
    <dependency>
      <groupId>org.example</groupId>
      <artifactId>a</artifactId>
      <version>${project.version}</version>
    </dependency>
    <dependency>
      <groupId>org.example</groupId>
      <artifactId>b</artifactId>
      <version>${project.version}</version>
    </dependency>
  </dependencies>

  <build>
    <plugins>
      <plugin>
        <groupId>org.jacoco</groupId>
        <artifactId>jacoco-maven-plugin</artifactId>
        <executions>
          <execution>
            <id>report</id>
            <phase>prepare-package</phase>
            <goals>
              <goal>report-aggregate</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>
</project>

and finally most important part - pom.xml where happens all the magic:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
    <modelVersion>4.0.0</modelVersion>

  <groupId>org.example</groupId>
  <artifactId>example</artifactId>
  <version>1.0-SNAPSHOT</version>
  <packaging>pom</packaging>

  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <maven.compiler.source>1.8</maven.compiler.source>
    <maven.compiler.target>1.8</maven.compiler.target>

    <jacoco.version>0.7.9</jacoco.version>
    <powermock.version>1.6.6</powermock.version>
  </properties>

  <modules>
    <module>a</module>
    <module>b</module>
  </modules>

  <dependencies>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>4.8.2</version>
      <scope>test</scope>
    </dependency>

    <dependency>
      <groupId>org.powermock</groupId>
      <artifactId>powermock-module-junit4</artifactId>
      <version>${powermock.version}</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.powermock</groupId>
      <artifactId>powermock-api-mockito</artifactId>
      <version>${powermock.version}</version>
      <scope>test</scope>
    </dependency>
  </dependencies>

  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-jar-plugin</artifactId>
        <version>3.0.2</version>
      </plugin>
    </plugins>
  </build>

  <profiles>
    <profile>
      <id>coverage</id>
      <modules>
        <module>report</module>
      </modules>
      <dependencies>
        <dependency>
          <groupId>org.jacoco</groupId>
          <artifactId>org.jacoco.agent</artifactId>
          <classifier>runtime</classifier>
          <version>${jacoco.version}</version>
          <scope>test</scope>
        </dependency>
      </dependencies>
      <build>
        <plugins>
          <plugin>
            <groupId>org.jacoco</groupId>
            <artifactId>jacoco-maven-plugin</artifactId>
            <version>${jacoco.version}</version>
            <executions>
              <execution>
                <id>instrument</id>
                <phase>process-classes</phase>
                <goals>
                  <goal>instrument</goal>
                </goals>
              </execution>
              <execution>
                <id>report</id>
                <!--
                restore original classes before generation of report,
                but after packaging of JAR:
                -->
                <phase>post-integration-test</phase>
                <goals>
                  <goal>restore-instrumented-classes</goal>
                  <goal>report</goal>
                </goals>
              </execution>
            </executions>
          </plugin>
          <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-surefire-plugin</artifactId>
            <version>2.12.2</version>
            <configuration>
              <systemPropertyVariables>
                <jacoco-agent.destfile>target/jacoco.exec</jacoco-agent.destfile>
              </systemPropertyVariables>
            </configuration>
          </plugin>
        </plugins>
      </build>
    </profile>
  </profiles>
</project>

in such setup

mvn clean verify -Pcoverage

will generate

• a/target/site/jacoco/index.html containing coverage for classes in module a by tests in module a
• b/target/site/jacoco/index.html containing coverage for classes in module b by tests in module b
• report/target/site/jacoco-aggregate/index.html containing coverage for classes in modules a and b by tests in modules a and b

but produced JAR files contain instrumented classes, however

mvn package -DskiptTests -Dmaven.jar.forceCreation

will produce non-instrumented JARs.

This complexity not needed in case if you get rid of PowerMockito. In this case coverage profile in pom.xml can be simplified to:

<profile>
  <id>coverage</id>
  <modules>
    <module>report</module>
  </modules>
  <build>
    <plugins>
      <plugin>
        <groupId>org.jacoco</groupId>
        <artifactId>jacoco-maven-plugin</artifactId>
        <version>${jacoco.version}</version>
        <executions>
          <execution>
            <goals>
              <goal>prepare-agent</goal>
              <goal>report</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>
</profile>

and mvn clean verify -Pcoverage will generate same reports and JARs that do not require rebuilding.