Periodic problem with parallel execution of tests

Question

Good afternoon! I use following stack for implementation automation tests: Java 8, Maven, Jenkins for automation execution of tests. From time to time (not every time, approximately 3-5% of all executions) I get problem with test(s) during their parallel execution. Parallel execution is provided by Jenkins and jenkins file. Example structure of jenkins file:

stage('First suite'){
            parallel {
                stage('1 test'){
                }
                stage('2 test'){
                }
            }
}

Last time I get following error:

[ERROR] java.lang.NullPointerException
[ERROR]     at java.util.Properties$LineReader.readLine(Properties.java:434)
[ERROR]     at java.util.Properties.load0(Properties.java:353)
[ERROR]     at java.util.Properties.load(Properties.java:341)
[ERROR]     at org.apache.maven.surefire.booter.SystemPropertyManager.loadProperties(SystemPropertyManager.java:50)
[ERROR]     at org.apache.maven.surefire.booter.BooterDeserializer.<init>(BooterDeserializer.java:62)
[ERROR]     at org.apache.maven.surefire.booter.ForkedBooter.setupBooter(ForkedBooter.java:109)
[ERROR]     at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:561)
[ERROR]     at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:548)

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M5:test (default-test) on project webuicheck: There are test failures.

[ERROR] Error occurred in starting fork, check output in log
[ERROR] Process Exit Code: 1
[ERROR] org.apache.maven.surefire.booter.SurefireBooterForkException: The forked VM terminated without properly saying goodbye. VM crash or System.exit called?

[ERROR]     at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:748)
[ERROR]     at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:305)
[ERROR]     at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:265)
[ERROR]     at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1314)
[ERROR]     at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:1159)
[ERROR]     at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:932)
[ERROR]     at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:137)
[ERROR]     at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:210)
[ERROR]     at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:156)
[ERROR]     at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:148)
[ERROR]     at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117)
[ERROR]     at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:81)
[ERROR]     at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:56)
[ERROR]     at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
[ERROR]     at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:305)
[ERROR]     at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:192)
[ERROR]     at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:105)
[ERROR]     at org.apache.maven.cli.MavenCli.execute(MavenCli.java:957)
[ERROR]     at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:289)
[ERROR]     at org.apache.maven.cli.MavenCli.main(MavenCli.java:193)
[ERROR]     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[ERROR]     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[ERROR]     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[ERROR]     at java.lang.reflect.Method.invoke(Method.java:498)
[ERROR]     at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:282)
[ERROR]     at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:225)
[ERROR]     at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:406)
[ERROR]     at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:347)

I've tried to follow all recommendation concerning:

• updating of maven-surefire-plugin (now I have one of the latest version);
• setting parameters in pom.xml:

                    <systemPropertyVariables>
                        <xmlOutputDir>${project.build.directory}/surefire</xmlOutputDir>
                    </systemPropertyVariables>
                    <classesDirectory>${project.build.outputDirectory}</classesDirectory>
                    <useSystemClassLoader>false</useSystemClassLoader>
                    <useManifestOnlyJar>false</useManifestOnlyJar>
                    <parallel>classes</parallel>
                    <forkCount>10</forkCount>
                    <reuseForks>true</reuseForks>
                    <useUnlimitedThreads>true</useUnlimitedThreads>
                    <argLine>-Xmx1024m -Xms64m</argLine>

I'm desperate and don't know what else to do. Maybe you've faced with the same problem during parallel execution in jenkins (single test execution always complete successfully). Thank you and have a nice day!

Answer 1

You should NOT run the Maven build twice in parallel. They will overwrite each other. This is very bad pattern.

Can you show me what is inside of the stages?

            stage('1 test'){
            }
            stage('2 test'){
            }

Answer 2

@tibor17, good day! Yes of course:

stage('First suite'){
   steps {
       catchError(buildResult: 'FAILURE', stageResult: 'FAILURE'){
          sh 'mvn -e -Dtest=FirstSuite test'
       }
    }
}
stage('Second suite'){
   steps {
       catchError(buildResult: 'FAILURE', stageResult: 'FAILURE'){
          sh 'mvn -e -Dtest=SecondSuite test'
       }
    }
}

But what did you mean 'overwrite each other'. Finally I get whole Allure report in the end of all stages where my mvn tests are executed and nothing is overwritten. The problem is: seldom (2-3% of all my launching) this leads to error and one or more tests/stages aren't executed and failed with maven-surefire-plugin error.

Answer 3

Good afternoon! I've generated effective POM:

<?xml version="1.0" encoding="Cp1251"?>
<!-- ====================================================================== -->
<!--                                                                        -->
<!-- Generated by Maven Help Plugin on 2021-07-15T12:23:27+03:00            -->
<!-- See: http://maven.apache.org/plugins/maven-help-plugin/                -->
<!--                                                                        -->
<!-- ====================================================================== -->
<!-- ====================================================================== -->
<!--                                                                        -->
<!-- Effective POM for project 'webuicheck:jar:1.0-SNAPSHOT'      -->
<!--                                                                        -->
<!-- ====================================================================== -->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/
xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId></groupId>
  <artifactId>webuicheck</artifactId>
  <version>1.0-SNAPSHOT</version>
  <properties>
    <allure.version>2.13.9</allure.version>
    <aspectj.version>1.9.7.M3</aspectj.version>
  </properties>
  <dependencies>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>4.12</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>com.codeborne</groupId>
      <artifactId>selenide</artifactId>
      <version>5.22.3</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>com.opencsv</groupId>
      <artifactId>opencsv</artifactId>
      <version>5.4</version>
      <scope>compile</scope>
    </dependency>
    <dependency>
      <groupId>io.qameta.allure</groupId>
      <artifactId>allure-selenide</artifactId>
      <version>2.13.9</version>
      <scope>compile</scope>
    </dependency>
    <dependency>
      <groupId>io.qameta.allure</groupId>
      <artifactId>allure-junit4</artifactId>
      <version>2.13.9</version>
      <scope>compile</scope>
    </dependency>
    <dependency>
      <groupId>javax.xml.bind</groupId>
      <artifactId>jaxb-api</artifactId>
      <version>2.3.0</version>
      <scope>compile</scope>
    </dependency>
    <dependency>
      <groupId>org.assertj</groupId>
      <artifactId>assertj-core</artifactId>
      <version>3.19.0</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>com.tngtech.junit.dataprovider</groupId>
      <artifactId>junit4-dataprovider</artifactId>
      <version>2.6</version>
      <scope>compile</scope>
    </dependency>
    <dependency>
      <groupId>org.slf4j</groupId>
      <artifactId>slf4j-nop</artifactId>
      <version>1.7.24</version>
      <scope>compile</scope>
    </dependency>
    <dependency>
      <groupId>org.apache.poi</groupId>
      <artifactId>poi-ooxml</artifactId>
      <version>4.1.2</version>
      <scope>compile</scope>
    </dependency>
    <dependency>
      <groupId>com.oracle.database.jdbc</groupId>
      <artifactId>ojdbc8</artifactId>
      <version>19.8.0.0</version>
      <scope>compile</scope>
    </dependency>
  </dependencies>
  <repositories>
    <repository>
      <snapshots>
        <enabled>false</enabled>
      </snapshots>
      <id>central</id>
      <name>Central Repository</name>
      <url>https://repo.maven.apache.org/maven2</url>
    </repository>
  </repositories>
  <pluginRepositories>
    <pluginRepository>
      <releases>
        <updatePolicy>never</updatePolicy>
      </releases>
      <snapshots>
        <enabled>false</enabled>
      </snapshots>
      <id>central</id>
      <name>Central Repository</name>
      <url>https://repo.maven.apache.org/maven2</url>
    </pluginRepository>
  </pluginRepositories>
  <build>
    <sourceDirectory>C:\Projects\webuicheck\src\main\java</sourceDirectory>
    <scriptSourceDirectory>C:\Projects\webuicheck\src\main\scripts</scriptSourceDirectory>
    <testSourceDirectory>C:\Projects\webuicheck\src\test\java</testSourceDirectory>
    <outputDirectory>C:\Projects\webuicheck\target\classes</outputDirectory>
    <testOutputDirectory>C:\Projects\webuicheck\target\test-classes</testOutputDirectory>
    <resources>
      <resource>
        <directory>C:\Projects\webuicheck\src\main\resources</directory>
      </resource>
    </resources>
    <testResources>
      <testResource>
        <filtering>true</filtering>
        <directory>C:\Projects\webuicheck\src\test\resources\properties</directory>
        <includes>
          <include>**/*.properties</include>
        </includes>
      </testResource>
    </testResources>
    <directory>C:\Projects\webuicheck\target</directory>
    <finalName>webuicheck-1.0-SNAPSHOT</finalName>
    <pluginManagement>
      <plugins>
        <plugin>
          <artifactId>maven-antrun-plugin</artifactId>
          <version>1.3</version>
        </plugin>
        <plugin>
          <artifactId>maven-assembly-plugin</artifactId>
          <version>2.2-beta-5</version>
        </plugin>
        <plugin>
          <artifactId>maven-dependency-plugin</artifactId>
          <version>2.8</version>
        </plugin>
        <plugin>
          <artifactId>maven-release-plugin</artifactId>
          <version>2.5.3</version>
        </plugin>
      </plugins>
    </pluginManagement>
    <plugins>
      <plugin>
        <artifactId>maven-compiler-plugin</artifactId>
        <version>3.8.1</version>
        <executions>
          <execution>
            <id>default-compile</id>
            <phase>compile</phase>
            <goals>
              <goal>compile</goal>
            </goals>
            <configuration>
              <source>1.8</source>
              <target>1.8</target>
            </configuration>
          </execution>
          <execution>
            <id>default-testCompile</id>
            <phase>test-compile</phase>
            <goals>
              <goal>testCompile</goal>
            </goals>
            <configuration>
              <source>1.8</source>
              <target>1.8</target>
            </configuration>
          </execution>
        </executions>
        <configuration>
          <source>1.8</source>
          <target>1.8</target>
        </configuration>
      </plugin>
      <plugin>
        <artifactId>maven-resources-plugin</artifactId>
        <version>3.0.2</version>
        <executions>
          <execution>
            <id>default-testResources</id>
            <phase>process-test-resources</phase>
            <goals>
              <goal>testResources</goal>
            </goals>
            <configuration>
              <encoding>UTF-8</encoding>
            </configuration>
          </execution>
          <execution>
            <id>default-resources</id>
            <phase>process-resources</phase>
            <goals>
              <goal>resources</goal>
            </goals>
            <configuration>
              <encoding>UTF-8</encoding>
            </configuration>
          </execution>
        </executions>
        <dependencies>
          <dependency>
            <groupId>org.apache.maven.shared</groupId>
            <artifactId>maven-filtering</artifactId>
            <version>3.1.1</version>
            <scope>compile</scope>
          </dependency>
        </dependencies>
        <configuration>
          <encoding>UTF-8</encoding>
        </configuration>
      </plugin>
      <plugin>
        <artifactId>maven-surefire-plugin</artifactId>
        <version>3.0.0-M5</version>
        <executions>
          <execution>
            <id>default-test</id>
            <phase>test</phase>
            <goals>
              <goal>test</goal>
            </goals>
            <configuration>
              <classesDirectory>C:\Projects\webuicheck\target\classes</classesDirectory>
              <useSystemClassLoader>false</useSystemClassLoader>
              <forkCount>4</forkCount>
              <reuseForks>false</reuseForks>
              <argLine>-Dfile.encoding=UTF-8 -javaagent:"${settings.localRepository}/org/aspectj/aspectjweaver/1.9.7.M3/aspectjweaver-1.9.7.M3.jar"</argLine>
              <properties>
                <property>
                  <name>listener</name>
                  <value>io.qameta.allure.junit4.AllureJunit4</value>
                </property>
              </properties>
              <systemProperties>
                <property>allure.results.directory</property>
                <value>target/allure-results</value>
              </systemProperties>
            </configuration>
          </execution>
        </executions>
        <dependencies>
          <dependency>
            <groupId>org.aspectj</groupId>
            <artifactId>aspectjweaver</artifactId>
            <version>1.9.7.M3</version>
            <scope>compile</scope>
          </dependency>
        </dependencies>
        <configuration>
          <classesDirectory>C:\Projects\webuicheck\target\classes</classesDirectory>
          <useSystemClassLoader>false</useSystemClassLoader>
          <forkCount>4</forkCount>
          <reuseForks>false</reuseForks>
          <argLine>-Dfile.encoding=UTF-8 -javaagent:"${settings.localRepository}/org/aspectj/aspectjweaver/1.9.7.M3/aspectjweaver-1.9.7.M3.jar"</argLine>
          <properties>
            <property>
              <name>listener</name>
              <value>io.qameta.allure.junit4.AllureJunit4</value>
            </property>
          </properties>
          <systemProperties>
            <property>allure.results.directory</property>
            <value>target/allure-results</value>
          </systemProperties>
        </configuration>
      </plugin>
      <plugin>
        <groupId>io.qameta.allure</groupId>
        <artifactId>allure-maven</artifactId>
        <version>2.10.0</version>
        <configuration>
          <reportVersion>2.13.9</reportVersion>
        </configuration>
      </plugin>
      <plugin>
        <artifactId>maven-clean-plugin</artifactId>
        <version>2.5</version>
        <executions>
          <execution>
            <id>default-clean</id>
            <phase>clean</phase>
            <goals>
              <goal>clean</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
      <plugin>
        <artifactId>maven-jar-plugin</artifactId>
        <version>2.4</version>
        <executions>
          <execution>
            <id>default-jar</id>
            <phase>package</phase>
            <goals>
              <goal>jar</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
      <plugin>
        <artifactId>maven-install-plugin</artifactId>
        <version>2.4</version>
        <executions>
          <execution>
            <id>default-install</id>
            <phase>install</phase>
            <goals>
              <goal>install</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
      <plugin>
        <artifactId>maven-deploy-plugin</artifactId>
        <version>2.7</version>
        <executions>
          <execution>
            <id>default-deploy</id>
            <phase>deploy</phase>
            <goals>
              <goal>deploy</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
      <plugin>
        <artifactId>maven-site-plugin</artifactId>
        <version>3.3</version>
        <executions>
          <execution>
            <id>default-site</id>
            <phase>site</phase>
            <goals>
              <goal>site</goal>
            </goals>
            <configuration>
              <outputDirectory>C:\Projects\webuicheck\target\site</outputDirectory>
              <reportPlugins>
                <reportPlugin>
                  <groupId>org.apache.maven.plugins</groupId>
                  <artifactId>maven-project-info-reports-plugin</artifactId>
                </reportPlugin>
              </reportPlugins>
            </configuration>
          </execution>
          <execution>
            <id>default-deploy</id>
            <phase>site-deploy</phase>
            <goals>
              <goal>deploy</goal>
            </goals>
            <configuration>
              <outputDirectory>C:\Projects\webuicheck\target\site</outputDirectory>
              <reportPlugins>
                <reportPlugin>
                  <groupId>org.apache.maven.plugins</groupId>
                  <artifactId>maven-project-info-reports-plugin</artifactId>
                </reportPlugin>
              </reportPlugins>
            </configuration>
          </execution>
        </executions>
        <configuration>
          <outputDirectory>C:\Projects\webuicheck\target\site</outputDirectory>
          <reportPlugins>
            <reportPlugin>
              <groupId>org.apache.maven.plugins</groupId>
              <artifactId>maven-project-info-reports-plugin</artifactId>
            </reportPlugin>
          </reportPlugins>
        </configuration>
      </plugin>
    </plugins>
  </build>
  <reporting>
    <outputDirectory>C:\Projects\webuicheck\target\site</outputDirectory>
  </reporting>
  <profiles>
    <profile>
      <id>firefox</id>
      <properties>
        <browser>firefox</browser>
      </properties>
    </profile>
    <profile>
      <id>chrome</id>
      <properties>
        <browser>chrome</browser>
      </properties>
    </profile>
    <profile>
      <id>ie</id>
      <properties>
        <browser>ie</browser>
      </properties>
    </profile>
  </profiles>
</project>

Answer 4

@stru4OK As I previously said the failure may be caused due to both stages are being overridden in the directory target/surefire. I think we can workaround this problem by setting "tempDir", see https://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#tempDir

If it would work for you we can confirm my hypothesis. So we have to give it a try.

stage('suites') {
        parallel {
            stage('1 test') {
              sh 'mvn clean test -DtempDir=stage1 -Dtest=FirstSuite'
            }
            stage('2 test') {
              sh 'mvn clean test -DtempDir=stage2 -Dtest=SecondSuite'
            }
        }
}

Answer 5

@tibor17, thanks for this suggestion! I'll check it. Also now I'm test another approach with different target directory for every thread. I define in pom.xml different target directory path:

 <profiles>
    <profile>
        <id>firstOutputDir</id>
        <build>
            <directory>${project.basedir}/target1/test-classes</directory>
        </build>
    </profile>

    <profile>
        <id>secondOutputDir</id>
        <build>
            <directory>${project.basedir}/target2/test-classes</directory>
        </build>
    </profile>
</profiles>

And use different profiles in executed suites:

        stage('Tests'){
        parallel {
            stage('FirstSuite'){
                steps {
                    sh 'mvn -e -Dtest=FirstSuite test -PfirstOutputDir'
                }
            }
            stage('SecondSuite'){
                steps {
                    sh 'mvn -e -Dtest=SecondSuite test -PsecondOutputDir'
                }
            }
        }
    }

What do you think about such approach? During last 15-20 execution I didn't face any surefire error. Maybe it's coincidence or maybe solution. Not sure)