Non-deterministic behavior of skipped tests in parallel testing on Maven Surefire

Question

I have a Maven project with the following expected results when tests are executed (mvn test):

Tests run: 543, Failures: 0, Errors: 0, Skipped: 8

I was exploring the benefit of running these tests in parallel to see if I could reduce the total elapsed time of the building process.

After adding the following setting to Surefire:

<parallel>methods</parallel>
<threadCount>5</threadCount>
<perCoreThreadCount>true</perCoreThreadCount>
<forkCount>3C</forkCount>
<reuseFork>true</reuseFork>

the number of Skipped tests vary. See the summary below for 50 executions:

01. Tests run: 543, Failures: 2, Errors: 176, Skipped: 8

... 34 runs with similar outputs...

35. Tests run: 543, Failures: 6, Errors: 173, Skipped: 8
36. **Tests run: 543, Failures: 2, Errors: 182, Skipped: 11**
37. **Tests run: 543, Failures: 2, Errors: 176, Skipped: 12**
38. Tests run: 543, Failures: 4, Errors: 177, Skipped: 8
39. Tests run: 543, Failures: 3, Errors: 175, Skipped: 8
40. Tests run: 543, Failures: 4, Errors: 174, Skipped: 8
41. Tests run: 543, Failures: 3, Errors: 172, Skipped: 8
42. Tests run: 543, Failures: 3, Errors: 172, Skipped: 8
43. **Tests run: 543, Failures: 2, Errors: 176, Skipped: 12**
44. Tests run: 543, Failures: 3, Errors: 176, Skipped: 8

... 5 runs with similar outputs...

50. Tests run: 543, Failures: 3, Errors: 172, Skipped: 8

Although it happened in 3/50 of the cases, I was expecting to see different numbers only in Errors and Failures but never in Skipped.

Any thoughts about the subject? Thanks in advance.

EDIT:

Junit Version: 4.12
Surefire Version: 2.18.1

@Tunaki I edited my post with JUnit and Surefire versions. They are 4.12 and 2.18.1 respectively. — Jeanderson Candido, Jun 01 '16 at 18:03
This looks like a bug in surefire. I cannot find an existing one though. Could you try with different `` values? — Tunaki, Jun 01 '16 at 18:25
So I discovered that tests can be dinamically ignored with JUnit. More details here: http://stackoverflow.com/questions/1689242/conditionally-ignoring-tests-in-junit-4 I assume this is probably related to what I observed. I'm going to see in details what happended with this extra skipped tests. Otherwise it could be a bug. — Jeanderson Candido, Jun 01 '16 at 19:02
Nice find, didn't know that either. Are you using assumptions in your test code? This could a valid explanation then. — Tunaki, Jun 01 '16 at 19:10
@Tunaki indeed... I just updated this thread with my answer. I searched for Assumptions and found some tests with it. — Jeanderson Candido, Jun 02 '16 at 00:40

score 2 · Accepted Answer · edited Jun 02 '16 at 10:29

FINDING:

Tests can be ignored in runtime execution:

Indeed, there are tests with Assumptions in this particular test suite:

Assume.assumeTrue(InetAddress.getAllByName(host).length > 1);
Assume.assumeNoException(x);
Assume.assumeTrue(sslContextFactory == null);
Assume.assumeNoException(x);
Assume.assumeTrue(OS.IS_LINUX);
Assume.assumeTrue(OS.IS_LINUX);
Assume.assumeTrue(!OS.IS_WINDOWS);
Assume.assumeTrue(!OS.IS_WINDOWS);
Assume.assumeThat(Integer.parseInt(nano), Matchers.greaterThan(21));
Assume.assumeTrue(false);
Assume.assumeTrue(true);
Assume.assumeTrue(false);

Given the circumstances, I believe the non-determinism is due to Assumptions that can be violated when running tests in parallel.

Non-deterministic behavior of skipped tests in parallel testing on Maven Surefire

1 Answers1