Maven Surefire Double-Counting Tests

Question

While not terribly well-versed in Maven lifecycles, I have read a fair amount of documentation over the last ~three days and found nothing showing a relationship between build cycles, goals, or my plugins and the issue I'm seeing.

I'm experiencing a weird issue with the latest Surefire plugin (3.0.0-M1) where my unit tests are completing successfully but appear to be running twice (or maybe are just being counted twice?).

Some context - before I specified any plugin or version in this POM file, Surefire ran on its own with a default version using:

mvn clean package

I have no idea why. And I never really stopped to ask why that might be. It just always has so I took it for granted and now it's behaving weird. I probably deserve this.

This is the POM file for that run (with some properties and distribution removed):

<build>
    <sourceDirectory>src/main/java</sourceDirectory>
    <plugins>
        <plugin>
            <artifactId>maven-compiler-plugin</artifactId>
            <version>3.5.1</version>
            <configuration>
                <source>1.8</source>
                <target>1.8</target>
            </configuration>
        </plugin>
        <plugin>
            <groupId>org.jacoco</groupId>
            <artifactId>jacoco-maven-plugin</artifactId>
            <version>0.7.8</version>
            <configuration>
                <destFile>${basedir}/target/jacoco.exec</destFile>
                <dataFile>${basedir}/target/jacoco.exec</dataFile>
            </configuration>
            <executions>
                <execution>
                    <id>jacoco-initialize</id>
                    <goals>
                        <goal>prepare-agent</goal>
                    </goals>
                </execution>
                <execution>
                    <id>jacoco-site</id>
                    <phase>test</phase>
                    <goals>
                        <goal>report</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-shade-plugin</artifactId>
            <version>3.0.0</version>
            <executions>
                <execution>
                    <phase>package</phase>
                    <goals>
                        <goal>shade</goal>
                    </goals>
                    <configuration>
                        <createDependencyReducedPom>true</createDependencyReducedPom>
                        <transformers>
                            <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                                <mainClass>com.capitalone.cardcore.App</mainClass>
                            </transformer>
                        </transformers>
                        <artifactSet>
                            <excludes>
                                <artifact>junit:*</artifact>
                                <artifact>org.hamcrest:*</artifact>
                                <artifact>org.junit.jupiter:*</artifact>
                                <artifact>org.apiguardian:*</artifact>
                                <artifact>org.powermock:*</artifact>
                                <artifact>org.mockito:*</artifact>
                                <artifact>commons-beanutils:*</artifact>
                                <artifact>javax.xml.*</artifact>
                                <artifact>org.apache.hadoop:hadoop-yarn-client:*</artifact>
                                <artifact>org.apache.hadoop:hadoop-yarn-server-common:*</artifact>
                                <artifact>org.apache.hadoop:hadoop-yarn-api:*</artifact>
                            </excludes>
                        </artifactSet>
                        <filters>
                            <filter>
                                <artifact>*:*</artifact>
                                <excludes>
                                    <exclude>META-INF/*.SF</exclude>
                                    <exclude>META-INF/*.DSA</exclude>
                                    <exclude>META-INF/*.RSA</exclude>
                                    <exclude>**/*.dll</exclude>
                                </excludes>
                            </filter>
                        </filters>
                    </configuration>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>
<dependencies>
    <dependency>
        <groupId>log4j</groupId>
        <artifactId>log4j</artifactId>
        <version>1.2.17</version>
    </dependency>
    <dependency>
        <groupId>com.google.code.gson</groupId>
        <artifactId>gson</artifactId>
        <version>2.8.1</version>
    </dependency>
    <dependency>
        <groupId>org.apache.commons</groupId>
        <artifactId>commons-collections4</artifactId>
        <version>4.1</version>
    </dependency>
    <dependency>
        <groupId>org.apache.commons</groupId>
        <artifactId>commons-lang3</artifactId>
        <version>3.6</version>
    </dependency>
    <dependency>
        <groupId>commons-io</groupId>
        <artifactId>commons-io</artifactId>
        <version>2.4</version>
    </dependency>
    <dependency>
        <groupId>org.apache.poi</groupId>
        <artifactId>poi-ooxml</artifactId>
        <version>3.9</version>
    </dependency>

    <!-- HTTP -->
    <dependency>
        <groupId>org.apache.httpcomponents</groupId>
        <artifactId>httpcore</artifactId>
        <version>4.4.4</version>
    </dependency>
    <dependency>
        <groupId>org.apache.httpcomponents</groupId>
        <artifactId>httpclient</artifactId>
        <version>4.5.2</version>
    </dependency>
    <dependency>
        <groupId>com.google.http-client</groupId>
        <artifactId>google-http-client</artifactId>
        <version>1.23.0</version>
    </dependency>
    <dependency>
        <groupId>com.google.http-client</groupId>
        <artifactId>google-http-client-jackson2</artifactId>
        <version>1.23.0</version>
    </dependency>

    <!-- JRECORD -->
    <dependency>
        <groupId>net.sf.JRecord</groupId>
        <artifactId>JRecord</artifactId>
        <version>0.81.1</version>
    </dependency>
    <dependency>
        <groupId>net.sf.bruce_a_martin.cb2xml</groupId>
        <artifactId>cb2xml</artifactId>
        <version>0.95.3</version>
    </dependency>

    <!-- PGP -->
    <dependency>
        <groupId>org.bouncycastle</groupId>
        <artifactId>bcpg-jdk15on</artifactId>
        <version>1.47</version>
    </dependency>
    <dependency>
        <groupId>org.bouncycastle</groupId>
        <artifactId>bcpkix-jdk15on</artifactId>
        <version>1.47</version>
    </dependency>

    <!-- UNIT TESTS -->
    <dependency>
        <groupId>junit</groupId>
        <artifactId>junit</artifactId>
        <version>4.11</version>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.junit.jupiter</groupId>
        <artifactId>junit-jupiter-api</artifactId>
        <version>5.0.3</version>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.powermock</groupId>
        <artifactId>powermock-core</artifactId>
        <version>${powermock.version}</version>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.powermock</groupId>
        <artifactId>powermock-api-mockito2</artifactId>
        <version>${powermock.version}</version>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.powermock</groupId>
        <artifactId>powermock-module-junit4</artifactId>
        <version>${powermock.version}</version>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.powermock</groupId>
        <artifactId>powermock-api-easymock</artifactId>
        <version>${powermock.version}</version>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.mockito</groupId>
        <artifactId>mockito-all</artifactId>
        <version>2.0.2-beta</version>
        <scope>test</scope>
    </dependency>

    <!-- PARQUET OUTPUT -->
    <dependency>
        <groupId>org.apache.parquet</groupId>
        <artifactId>parquet-avro</artifactId>
        <version>${parquet.version}</version>
    </dependency>
    <dependency>
        <groupId>org.apache.parquet</groupId>
        <artifactId>parquet-common</artifactId>
        <version>${parquet.version}</version>
    </dependency>
    <dependency>
        <groupId>org.apache.parquet</groupId>
        <artifactId>parquet-encoding</artifactId>
        <version>${parquet.version}</version>
    </dependency>
    <dependency>
        <groupId>org.apache.parquet</groupId>
        <artifactId>parquet-column</artifactId>
        <version>${parquet.version}</version>
    </dependency>
    <dependency>
        <groupId>org.apache.parquet</groupId>
        <artifactId>parquet-hadoop</artifactId>
        <version>${parquet.version}</version>
    </dependency>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-client</artifactId>
        <version>${hadoop.version}</version>
    </dependency>
</dependencies>

So - no mention of the surefire plugin, but it ran my tests anyway. Runs all the tests and gives me back 144 as the count (which is correct). It's been like this for over a year, but I added a few code tweaks and now I need to include it as a plugin because I need a few environment variables for my tests like so:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>3.0.0-M1</version>
        <configuration>
            <environmentVariables>
                <env>dev</env>
                <ENV>DEV</ENV>
            </environmentVariables>
        </configuration>
</plugin>

But now I get this when I run my test suite:

The first question is: am I losing my mind? When I don't include the plugin, the surefire plugin is running by default when I do nothing (I don't even know how to turn it off), and when I include it, it appears to run twice. Is one of my other plugins running it without broadcasting it? That's a question I'm not proud of. But then - if I keep the plugin included in my POM as a line-item but set it to that "default" version:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>2.12.4</version>
        <configuration>
            <environmentVariables>
                <env>dev</env>
                <ENV>DEV</ENV>
            </environmentVariables>
        </configuration>
</plugin>

I get my expected 144 count back and my tests run because they have the variables they need. So is there maybe a problem with the latest version of surefire? Or just the way I'm including it with other packages?

Anyone encountered this with this new Surefire? Or maybe this combination of plugins?

EDIT #1 - 11/18/2018

Including my TestSuite class:

package com.business.cardcore.tests;

import com.business.cardcore.tests.classes.app.*;
import com.business.cardcore.tests.classes.librarytools.*;
import org.junit.runner.RunWith;
import org.junit.runners.Suite;

@RunWith(Suite.class)

// run the test classes
@Suite.SuiteClasses({
        // App
        AppCoreTests.class,
        AppFileMapperErrorTests.class,
        AppFileProcessorErrorTests.class,
        AppSchemaWriterErrorTests.class,
        // LibraryTools
        ChamberOfSecretsTests.class,
        ConfigurationTests.class,
        CopybookReaderErrorTests.class,
        CopybookReaderTests.class,
        DatawiseTests.class,
        FileMapperTests.class,
        HttpWrapperTests.class,
        OutputAsciiTests.class,
        OutputEbcdicTests.class,
        OutputParquetTests.class,
        PGPDecryptTests.class,
        SchemaWriterTests.class,
        TokenValidationTests.class,
        TuringTests.class,
        ValidationTests.class
})

public class TestSuite { }

Its very strange. I tried replicating on my own project but couldn't . I changed my project to use the same compiler, jacoco, and surefire plugin versions. The only plugin I dont have on my project is the maven-shade-plugin. If you think its a plugin combo, what happens when you exclude the maven-shade-plugin from your project? Also, what version of Maven are you using? — John Camerin, Nov 17 '18 at 02:32
To run junit-jupiter you need at least surefire 2.22.1 otherwise you will not be able to run it...furthermore you need to add the correct versions for junit5 ..`junit-jupiter-engine`...Furthermoe you should show your tests.... — khmarbaise, Nov 17 '18 at 07:50
don't know what the problem with running twice is. But when you run `mvn package` maven will implicity run _compile_, _test_ and then _package_. That's default behavior. — P.J.Meisch, Nov 17 '18 at 16:59
@khmarbaise I'm editing my question to show a screenshot of my test suite - I inherited this pattern from another developer and didn't question it because it ran (there is that taking it for granted...) @P.J.Meisch I phrased my question poorly - the better question is: Knowing that _test_ is run by default using the package command, how does Maven decide which version of Surefire to use if I never specify one? Maven version: `mvn --version Apache Maven 3.5.0 (ff8f5e7444045639af65f6095c62210b5713f426; 2017-04-03T15:39:06-04:00)` — mzogz, Nov 18 '18 at 15:37
@khmarbaise I attempted to update everything to JUnit 5 via the documentation found here: https://junit.org/junit5/docs/current/user-guide/ but I ran into more issues with PowerMock than I had patience to solve in an otherwise working test environment. I will definitely spend more time on it after this release. I've update my JUnit to 4.12, which is the base jar that supports that updated Junit Jupiter implementation. Thank you for pointing that out! — mzogz, Nov 19 '18 at 15:15

score 1 · Accepted Answer · answered Nov 17 '18 at 20:38

1

See the References section of this answer to Maven: Lifecycle vs. Phase vs. Plugin vs. Goal for why plugins run during a Maven build without having declared them.

A wild guess concerning your doubled tests: In your first image we can see that you're running a TestSuite. Maybe your tests are run (or counted) once via this suite and once individually. What if you remove/disable the suite.

answered Nov 17 '18 at 20:38

Gerold Broser

14,080
5
48
107

Removing the TestSuite with the updated version of Maven Surefire caused the tests to only be counted once while still running everything. In hindsight, I'm not sure why the original developer created the TestSuite in addition to the tests. – mzogz Nov 19 '18 at 15:14
@mzogz Using `TestSuite`s is absolutely OK. [They aggregate tests](https://github.com/junit-team/junit4/wiki/Aggregating-tests-in-suites) so that you don't have to run them individually. They work here with _Surefire Plugin 2.22.1_. Perhaps the _3.0.0-M1_ milestone is a few yards off the intended target. – Gerold Broser Nov 19 '18 at 15:44

Maven Surefire Double-Counting Tests

1 Answers1