maven-failsafe-plugin Failures and BUILD SUCCESS?

Question

I want to use maven-failsafe-plugin to run some integration tests. If any test fails, I want Maven to fail the build and not BUILD SUCCESS.

Tests run: 103, Failures: 1, Errors: 0, Skipped: 26
[INFO] BUILD SUCCESS*

how can I configure it, that build not success is?

My failsafe plugin is configured as:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-failsafe-plugin</artifactId>
    <version>${failsafe.version}</version>
    <configuration>
        <systemProperties>
            <CI_INTEGRATION_OVERRIDE_PATH>${basedir}/..</CI_INTEGRATION_OVERRIDE_PATH>
        </systemProperties>
        <includes>
            <include>**/integration/**/*.java</include>
        </includes>
        <excludes>
            <exclude>**/integration/**/*TestSuite.java</exclude>
        </excludes>
    </configuration>
    <executions>
        <execution>
            <id>integration-test</id>
            <goals>
                <goal>integration-test</goal>
                <goal>verify</goal>
            </goals>
        </execution>
    </executions>
</plugin>

Which version of maven-failsafe-plugin do you use? Which Maven version? How did you called mvn to run integration tests? — khmarbaise, Sep 05 '12 at 11:08
See [this mail thread](http://maven.40175.n5.nabble.com/Failing-a-build-with-maven-failsafe-plugin-td3199308.html) on "Maven - Users". — Andrew Logvinov, Sep 05 '12 at 11:11
MAVEN CALL: mvn clean install -P jars failsafe:integration-test -e 2.12 MAVEN 3.16 — Fawi, Sep 05 '12 at 11:23
If i call mvn `failsafe:verify` then throws an error `cannot find failsafe-summary.xml` — Fawi, Sep 06 '12 at 09:07

score 17 · Answer 1 · edited Nov 07 '18 at 10:43

17

As Andrew pointed out, the correct solution is to use failsafe as intended. The integration-test goal is specifically designed not to fail the build. If you want to fail the build, call mvn verify or mvn failsafe:verify

For the verify goal to work, it needs to read the results of the integration test which by default are written to ${project.build.directory}/failsafe-reports/failsafe-summary.xml so make sure that is getting generated.

Also, you have to make sure you bind your maven-failsafe-plugin configuration to both the integration-test goal and the verify goal in the executions part.

Failure to add either of those will lead to maven succeeding the build instead of failing it when integration tests fail.

edited Nov 07 '18 at 10:43

Adam

5,215
5
51
90

answered Oct 02 '13 at 13:36

Ben Mathews

2,939
2
19
25

1

I do agree with your answer. The it-test goal is by design doesn't fail. "My goals were different and I wanted to make an opinionated piece of software and I preferred the notion of convention over configuration." as quoted by Jason van Zyl in the history of maven article. It is always good to follow the conventions rather than hacking around to do stuff which it is not designed for. – NewUser Aug 25 '16 at 08:54
3

I often find that the intents of the Maven authors appear to directly contradict the way a sane person might attempt to use it. – cbmanica Feb 01 '17 at 01:17
`mvn verify` will succeed even if the integration tests fail - `Tests run: 5, Failures: 1` --> `BUILD SUCCESS`. If you remove your last sentence, I'll remove my down-vote. – Adam Nov 07 '18 at 10:24
Ben I discovered how it works. The `maven-failsafe-plugin` must have the `summary-file` parameter in its config in order for the `mvn verify` phase to fail - so you were mostly correct but lacked a key detail. I'll edit your answer. – Adam Nov 07 '18 at 10:34

score 2 · Answer 2 · edited May 23 '17 at 12:09

2

Since you are running mvn clean install both the integration-test and verify phases should be executing. According to the failsafe plugin docs the failsafe:integration-test and failsafe:verify goals are bound to those phases, so I don't believe the extra call to failsafe:integration-test is required.

That said however, I'm not sure I trust the failsafe plugin documentation. I answered a similar question for someone earlier this year. It turned out he had to explicitly bind each goal to the correct phase and then failsafe worked as expected. Might be worth a shot.

edited May 23 '17 at 12:09

Community

1
1

answered Sep 05 '12 at 13:36

user944849

14,524
2
61
83

I have found a solution, i can use maven-surefire-plugin for my integration tests. – Fawi Oct 01 '12 at 09:10

score 0 · Answer 3 · answered May 17 '22 at 08:55

I had a similar issue with Jenkins builds.

Running the tests locally resulted in failed tests and thus failed build
Running the tests on Jenkins resulted in failed tests but "BUILD SUCCESS"

Solution: in job configuration set MAVEN_OPTS to -Dmaven.test.failure.ignore=false.

Jenkins by default seems to ignore failed integration tests.

See also https://stackoverflow.com/a/28684048/4412885

score -1 · Accepted Answer · answered Oct 04 '12 at 08:00

solution.

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-surefire-plugin</artifactId>
  <configuration>
    <skip>true</skip>
  </configuration>
  <executions>
    <execution>
      <id>unit-test</id>
      <phase>test</phase>
      <goals>
        <goal>test</goal>
      </goals>
      <configuration>
        <skip>false</skip>
        <excludes>
          <exclude>**/*IntegrationTest.java</exclude>
        </excludes>
      </configuration>
      </execution>
      <execution>
        <id>integration-test</id>
        <phase>integration-test</phase>
        <goals>
          <goal>test</goal>
        </goals>
        <configuration>
          <skip>false</skip>
          <enableAssertions>false</enableAssertions>
          <includes>
            <include>**/*IntegrationTest.java</include>
          </includes>
          <systemPropertyVariables>
            <integration>${integration}</integration>
          </systemPropertyVariables>
        </configuration>
      </execution>
    </executions>
</plugin>

This answer is replacing the failsafe plugin with the surefire plugin. — Andrew, May 08 '13 at 15:42
If your integration testing doesn't require pre- and post- phases (set up and tear down) then use `surefire` - if you want to stop a server or delete a database, you shouldn't use a plugin that will fail since the `post-integration-test` phase then won't fire. — Adam, Nov 07 '18 at 10:26
The question is how to make failsafe-plugin fail the build if there are test failures. "use surefire instead" does not answer the question. — mihca, May 16 '22 at 06:47

maven-failsafe-plugin Failures and BUILD SUCCESS?

4 Answers4

Linked