Why does pitest work with maven on the commandline, but not in Jenkins-Pipeline?

Question

I have a strange problem here for which I could not find the reason:

When using pitest with maven (and java 11) on the commandline it works as expected, but when running it on the Jenkins-Server via declarative pipeline it results in an error.

Part from my maven pom:

<build>
  <plugins>
    <plugin>
      <groupId>org.pitest</groupId>
      <artifactId>pitest-maven</artifactId>
      <version>1.4.8</version>
      <dependencies>
        <dependency>
          <groupId>org.pitest</groupId>
          <artifactId>pitest-junit5-plugin</artifactId>
          <version>0.8</version>
        </dependency>
      </dependencies>
      <configuration>
        <verbose>true</verbose>
      </configuration>
    </plugin>
  </plugins>
</build>

Part from my Jenkinsfile:

stage('MutationTesting')
 {
  steps
   {
    bat 'mvn --batch-mode org.pitest:pitest-maven:mutationCoverage'
   }
 }

Before that the following maven commands will be called from the pipeline:

mvn --batch-mode clean

mvn --batch-mode compile

mvn --batch-mode compiler:testCompile surefire:test -Dmaven.test.failure.ignore=true

When calling exactly these commands on the commandline then pitest works perfectly.

When comparing the logging results, then they are nearly equal.

Log output from commandline:

PIT >> FINE : Maximum available memory is 3609 mb

PIT >> FINE : MINION : Installing PIT agent

PIT >> INFO : Sending 13 test classes to minion

PIT >> INFO : Sent tests to minion

PIT >> INFO : MINION : 08:51:20 PIT >> FINE : Expecting 13 tests classes from parent

PIT >> FINE : Tests classes received

PIT >> INFO : Checking environment

PIT >> INFO : MINION : 08:51:20 PIT >> INFO : Found 1 tests

PIT >> INFO : MINION : 08:51:20 PIT >> INFO : Dependency analysis reduced number of potential tests by 0

PIT >> INFO : MINION : 08:51:20 PIT >> INFO : 1 tests received

PIT >> INFO : MINION : 08:51:20 PIT >> FINE : Running 1 units

PIT >> INFO : MINION : 08:51:20 PIT >> FINE : Gathering coverage for test Description [testClass=de.test.Tests, name=creation()]

PIT >> FINE : Coverage generator Minion exited ok

PIT >> INFO : Calculated coverage in 0 seconds.

PIT >> FINE : Used memory after coverage calculation 42 mb

PIT >> FINE : Free Memory after coverage calculation 201 mb

[... Mutations output cut]

Log output from Jenkins-Pipeline:

PIT >> FINE : Maximum available memory is 1813 mb

PIT >> FINE : MINION : Installing PIT agent

PIT >> INFO : Sending 13 test classes to minion

PIT >> INFO : Sent tests to minion

PIT >> INFO : MINION : 08:46:28 PIT >> FINE : Expecting 13 tests classes from parent

PIT >> FINE : Tests classes received

PIT >> INFO : Checking environment

PIT >> INFO : MINION : 08:46:29 PIT >> INFO : Found 1 tests

PIT >> INFO : MINION : 08:46:29 PIT >> INFO : Dependency analysis reduced number of potential tests by 0

PIT >> INFO : MINION : 08:46:29 PIT >> INFO : 1 tests received

PIT >> INFO : MINION : 08:46:29 PIT >> FINE : Running 1 units

PIT >> INFO : MINION : 08:46:29 PIT >> FINE : Gathering coverage for test Description [testClass=de.test.Tests, name=creation()]

PIT >> FINE : Coverage generator Minion exited ok PIT >> INFO : Calculated coverage in 1 seconds. PIT >> FINE : Used memory after coverage calculation 43 mb PIT >> FINE : Free Memory after coverage calculation 177 mb PIT >> INFO : Created 0 mutation test units [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 3.644 s [INFO] Finished at: 2019-06-06T08:46:29+02:00 [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.pitest:pitest-maven:1.4.8:mutationCoverage (default-cli) on project multiuploader: Execution default-cli of goal org.pitest:pitest-maven:1.4.8:mutationCoverage failed: No mutations found. This probably means there is an issue with either the supplied classpath or filters. [ERROR] See http://pitest.org for more details.

So I could only assume that there is a difference in the environment - but which one? Or is it is a bug in pitest?

score 1 · Answer 1 · answered Jun 09 '19 at 14:33

1

The most likely issue is that pitest is being executed by jenkins in an environment where the tests have not been compiled.

The most robust approach is to bind pitest to the test phase in a profile rather than invoke the goal directly.

<profile>
  <id>pitest</id>
    <build>
      <plugins>
        <plugin>
          <groupId>org.pitest</groupId>
          <artifactId>pitest-maven</artifactId>
          <version>1.4.8</version>
          <executions>
            <execution>
              <id>pitest</id>
              <phase>test</phase>
              <goals>
                <goal>mutationCoverage</goal>
              </goals>
            </execution>
          </executions>  
        </plugin>
      </plugins>

Pitest can then be run with

mvn -Ppitest test

answered Jun 09 '19 at 14:33

henry

5,923
29
46

Could you please explain in more detail why pitest is executed in an environment where the tests have not been compiled? As I described my pipeline has compiled the tests already. – PowerStat Jun 09 '19 at 18:45
I can't as (if this is what is happening) it is an issue with the environment/jenkins. – henry Jun 10 '19 at 16:00
I am sure that the tests have been compiled in the jenkins environment - for more please have a look at my Question https://stackoverflow.com/questions/55183989/maven-lifecycle-within-jenkins-pipeline-how-to-best-separate-responsibilities This proves it, because I run tools like SpotBugs/FindBugs that also require the compiled code. – PowerStat Jun 11 '19 at 07:25

Why does pitest work with maven on the commandline, but not in Jenkins-Pipeline?

1 Answers1