How to run parallel Cucumber Spring Boot integration tests with Maven (surefire or failsafe)?

Question

Hello folks,
I'm developing cucumber tests since january of this year and i'm now facing a problem running cucumber integration tests in parallel with maven (surefire as well as failsafe) in a Spring Boot environment.

In a nutshell:
I want to run cucumber integration tests in parallel with a spring boot application running. I use maven with either the Surefire or Failsafe plugin active. Everything works fine when i comment out every Spring Boot component in the code. Then i have as many threads as feature-files and they run parallel. But when I add the Spring Boot components I need, the tests run serial.

My dependencies in the pom.xml:

<dependencies>
    <dependency>
        <groupId>io.cucumber</groupId>
        <artifactId>cucumber-java</artifactId>
        <version>${cucumber.version}</version>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>io.cucumber</groupId>
        <artifactId>cucumber-java8</artifactId>
        <version>${cucumber.version}</version>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>io.cucumber</groupId>
        <artifactId>cucumber-junit</artifactId>
        <version>${cucumber.version}</version>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>io.cucumber</groupId>
        <artifactId>cucumber-spring</artifactId>
        <version>${cucumber.version}</version>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.projectlombok</groupId>
        <artifactId>lombok</artifactId>
        <version>1.18.10</version>
        <scope>provided</scope>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-web</artifactId>
        <version>${spring.boot.version}</version>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-data-jpa</artifactId>
        <version>${spring.boot.version}</version>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-test</artifactId>
        <version>${spring.boot.version}</version>
        <scope>compile</scope>
    </dependency>
    <dependency>
        <groupId>org.slf4j</groupId>
        <artifactId>slf4j-api</artifactId>
        <version>1.7.30</version>
    </dependency>
    <dependency>
        <groupId>com.h2database</groupId>
        <artifactId>h2</artifactId>
        <version>1.4.200</version>
        <scope>test</scope>
    </dependency>
</dependencies>

My plugins in the pom.xml:

<plugins>
    <!-- Maven Compiler -->
    <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-compiler-plugin</artifactId>
        <version>3.8.1</version>
        <configuration>
            <source>8</source>
            <target>8</target>
        </configuration>
    </plugin>
    <!-- Option 1 to run cucumber tests in parallel: Surefire -->
    <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <version>2.22.0</version>
        <configuration>
            <parallel>methods</parallel>
            <useUnlimitedThreads>true</useUnlimitedThreads>
        </configuration>
    </plugin>
    <!-- Option 2 to run cucumber tests in parallel: Failsafe -->
    <!--<plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-failsafe-plugin</artifactId>
        <version>3.0.0-M4</version>
        <executions>
            <execution>
                <goals>
                    <goal>integration-test</goal>
                    <goal>verify</goal>
                </goals>
                <configuration>
                    <parallel>methods</parallel>
                    <threadCount>5</threadCount>
                    <useUnlimitedThreads>false</useUnlimitedThreads>
                    &lt;!&ndash; important for the right execution &ndash;&gt;
                    <perCoreThreadCount>true</perCoreThreadCount>
                </configuration>
            </execution>
        </executions>
    </plugin>-->
</plugins>

My Java classes:

RunCucumberIT.java --> execution with failsafe
RunCucumberTest.java --> same code as RunCucumberIT.java, other name for execution with surefire
StepDefinitions.java
TestApplication.java
CucumberSpringContextConfiguration.java

The Java code:

RunCucumberIT.java and RunCucumberTest.java

package parallel;
import io.cucumber.junit.Cucumber;
import io.cucumber.junit.CucumberOptions;
import org.junit.runner.RunWith;
/**
 * Class to run the cucumber tests with Surefire.
 */
@RunWith(Cucumber.class)
@CucumberOptions(plugin = { }, strict = true, snippets = CucumberOptions.SnippetType.CAMELCASE,
        features = "src/test/java/parallel", stepNotifications = true)
public class RunCucumberTest {
}

StepDefinitions.java

package parallel;
import io.cucumber.java8.En;
import lombok.extern.slf4j.Slf4j;
@Slf4j
public class StepDefinitions implements En {
    public StepDefinitions() {
        Given("Step from {string} in {string} feature file", this::printThreads);
    }
    private void printThreads(final String scenario, final String file) throws InterruptedException {
        log.info(" Thread ID - {} - {} from {} feature file.", Thread.currentThread().getId(), scenario, file);
        Thread.sleep(5000L);
    }
}

TestApplication.java

package parallel;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
@SpringBootApplication
public class TestApplication {
    public static void main(final String[] args) {
        SpringApplication.run(TestApplication.class, args);
    }
}

CucumberSpringContextConfiguration.java

package parallel;
import io.cucumber.java.Before;
import io.cucumber.spring.CucumberContextConfiguration;
import lombok.extern.slf4j.Slf4j;
import org.springframework.boot.test.context.SpringBootContextLoader;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.test.context.ContextConfiguration;
/**
 * Class to use spring application context while running cucumber.
 */
@Slf4j
@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
@ContextConfiguration(classes = TestApplication.class, loader = SpringBootContextLoader.class)
@CucumberContextConfiguration
public class CucumberSpringContextConfiguration {
    /**
     * Need this method so the cucumber will recognize this class as glue and load spring context configuration.
     */
    @Before
    public void setUp() {
        log.info("Spring Context initialized for executing cucumber tests.");
    }
}

My gherkin code:

scenario-outline.feature

Feature: Scenario Outlines feature file
  Scenario Outline: <scen_out_row_num>
    Given Step from '<scen_out_row_num>' in 'scenario-outlines' feature file
    Examples:
      | scen_out_row_num       |
      | Scenario Outline Row 1 |
      | Scenario Outline Row 2 |

scenarios-0.feature

Feature: Scenarios feature file
  Scenario: Scenario Number One -
    Given Step from 'Scenario 1' in 'scenarios-0' feature file
  Scenario: Scenario Number Two -
    Given Step from 'Scenario 2' in 'scenarios-0' feature file

scenarios-1.feature

Feature: Scenarios feature file
  Scenario: Scenario Number One a
    Given Step from 'Scenario 1' in 'scenarios-1' feature file
  Scenario: Scenario Number Two a
    Given Step from 'Scenario 2' in 'scenarios-1' feature file

scenarios-2.feature

Feature: Scenarios feature file
  Scenario: Scenario Number One b
    Given Step from 'Scenario 1' in 'scenarios-2' feature file
  Scenario: Scenario Number Two b
    Given Step from 'Scenario 2' in 'scenarios-2' feature file

scenarios-3.feature

Feature: Scenarios feature file
  Scenario: Scenario Number One c
    Given Step from 'Scenario 1' in 'scenarios-3' feature file
  Scenario: Scenario Number Two c
    Given Step from 'Scenario 2' in 'scenarios-3' feature file

Expected behaviour:
One thread per feature-file and parallel execution of the integration tests. Some output like that (this is the output when i comment out all Spring Boot stuff):

[INFO] --- maven-failsafe-plugin:3.0.0-M4:integration-test (default) @ cucumber-test ---
[INFO] 
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running Scenario Number One b
[pool-1-thread-4] INFO parallel.StepDefinitions -  Thread ID - 17 - Scenario 1 from scenarios-2 feature file.
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.338 s - in Scenario Number One b
[INFO] Running Scenario Number One a
[pool-1-thread-3] INFO parallel.StepDefinitions -  Thread ID - 16 - Scenario 1 from scenarios-1 feature file.
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.353 s - in Scenario Number One a
[INFO] Running Scenario Outline Row 1
[pool-1-thread-1] INFO parallel.StepDefinitions -  Thread ID - 14 - Scenario Outline Row 1 from scenario-outlines feature file.
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.363 s - in Scenario Outline Row 1
[INFO] Running Scenario Number One c
[pool-1-thread-5] INFO parallel.StepDefinitions -  Thread ID - 18 - Scenario 1 from scenarios-3 feature file.
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.371 s - in Scenario Number One c
[INFO] Running Scenario Number One -
[pool-1-thread-2] INFO parallel.StepDefinitions -  Thread ID - 15 - Scenario 1 from scenarios-0 feature file.
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.378 s - in Scenario Number One -
[INFO] Running Scenario Outline Row 2
[pool-1-thread-1] INFO parallel.StepDefinitions -  Thread ID - 14 - Scenario Outline Row 2 from scenario-outlines feature file.
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.003 s - in Scenario Outline Row 2
[INFO] Running Scenario Number Two a
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.011 s - in Scenario Number Two a
[INFO] Running Scenario Number Two c
[pool-1-thread-3] INFO parallel.StepDefinitions -  Thread ID - 16 - Scenario 2 from scenarios-1 feature file.
[pool-1-thread-5] INFO parallel.StepDefinitions -  Thread ID - 18 - Scenario 2 from scenarios-3 feature file.
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.017 s - in Scenario Number Two c
[INFO] Running Scenario Number Two b
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.027 s - in Scenario Number Two b
[INFO] Running Scenario Number Two -
[pool-1-thread-4] INFO parallel.StepDefinitions -  Thread ID - 17 - Scenario 2 from scenarios-2 feature file.
[pool-1-thread-2] INFO parallel.StepDefinitions -  Thread ID - 15 - Scenario 2 from scenarios-0 feature file.
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.043 s - in Scenario Number Two -


10
 Scenarios (
10 passed
)


10
 Steps (
10 passed
)


0m
10,486s

Actual behaviour:
Just one thread is used and all tests run serially. Output:

[INFO] --- maven-failsafe-plugin:3.0.0-M4:integration-test (default) @ cucumber-test ---
...    
  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::        (v2.2.4.RELEASE)
...
Thread ID - 1 - Scenario Outline Row 1 from scenario-outlines feature file.
2020-06-10 12:04:02.519  INFO 13028 --- [           main] o.s.t.c.support.AbstractContextLoader    : Could not detect default resource locations for test class [parallel.CucumberSpringContextConfiguration]: no resource found for suffixes {-context.xml, Context.groovy}.
...
2020-06-10 12:04:02.544  INFO 13028 --- [           main] parallel.StepDefinitions                 : 
... 
Thread ID - 1 - Scenario Outline Row 2 from scenario-outlines feature file.
2020-06-10 12:04:07.550  INFO 13028 --- [           main] o.s.t.c.support.AbstractContextLoader    : Could not detect default resource locations for test class [parallel.CucumberSpringContextConfiguration]: no resource found for suffixes {-context.xml, Context.groovy}.
Thread ID - 1 - Scenario 1 from scenarios-0 feature file.
...
Thread ID - 1 - Scenario 2 from scenarios-0 feature file.
...
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0

I hope this problem is understandable and someone can help me!

Thanks in advance,
Maximotus

You may want to redo everything with `stepNotifications = false` . The `stepNotifications` confuse surefire/failsafe in lots of different ways. — M.P. Korstanje, Jun 10 '20 at 13:05
When using `@CucumberContextConfiguration` you don't need the dummy step definition any more. — M.P. Korstanje, Jun 10 '20 at 13:25
@M.P.Korstanje Thanks for that hint, but it results in the same actual output (for both failsafe and surefire). I think the problem is Spring Boot that somehow obstructs multi-threading with its context or something. — maximotus, Jun 10 '20 at 13:30
@M.P.Korstanje Yeah you are right. I removed it. But still the same problem. Just the lines with "INFO 13028 --- [ main] p.CucumberSpringContextConfiguration : Spring Context initialized for executing cucumber tests." are missing in the output, as expected. — maximotus, Jun 10 '20 at 13:34
Beats me. I wrote tests to confirm it works a while ago. You'll have to deep dive. — M.P. Korstanje, Jun 10 '20 at 15:52
@AnchikaAgarwal Not yet. If I make any progress, I'll let you know! I think the problem is somewhere in SpringBOOT, because it works with Spring. — maximotus, Jun 17 '20 at 07:25
@maximotus I found the issue for me, not sure if its relevant to you. Cucumber was expecting junit4 but I had latest dependencies of junit5. So I had to add the vintage cucumber dependency post which it was able to run. I was able to run with surefie, but failsafe plugin seems to have some issue. Also, I have not yet able to run in parallel though. — Anchika Agarwal, Jun 17 '20 at 09:14
I've found out, that the dependency cucumber-junit includes junit4 for compile scope and junit5 (jupiter) and vintage for test. I've tested the follwing with a cucumber spring (not springBOOT!) project: I've used some jupiter and some junit4 classes and now I've changed every jupiter class to junit4. Then I've removed the redundant dependency junit4 from my pom.xml. And I've removed jupiter and vintage from my pom.xml. Now, the tests run in parallel. As soon as I include jupiter and vintage, the tests run serially. I'll try to use Spring boot and exclude jupiter and vintage next. — maximotus, Jun 22 '20 at 16:07

score 2 · Answer 1 · answered Oct 26 '20 at 11:17

For running cucumber test in parallel, you have to add in the pom.xml the exclusions for juni-jupiter and junit-vintage. This is how it worked for me:

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
            <exclusions>
                <exclusion>
                    <groupId>org.junit.jupiter</groupId>
                    <artifactId>junit-jupiter</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.junit.vintage</groupId>
                    <artifactId>junit-vintage-engine</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

How to run parallel Cucumber Spring Boot integration tests with Maven (surefire or failsafe)?

1 Answers1