0

We have a few hundreds of tests in our project, and we now need to run them all with a custom test runner. We have written a custom test runner, called RetryRunner, that retries a failing test for a configurable maximum number of attempts. The code for RetryRunner was taken from here: How to Re-run failed JUnit tests immediately?

Now, to run a test class with this RetryRunner, we either have the option of:

1.

@RunWith(CustomSuite.class)

@Suite.SuiteClasses({ TestClass1.class, TestClass2.class })

where CustomSuite.class returns our RetryRunner, so that the test retry mechanism is applied on TestClass1 and TestClass2. (Note: Some of our tests are indeed a part of @Suite.SuiteClasses)

OR

2.

@RunWith(RetryRunner.class) public class SampleTestClass {...}

which will enable all tests in SampleTestClass to be run with RetryRunner. And we similarly keep using @RunWith(RetryRunner.class) for every test class.

Now, although we would have loved to keep all our tests within @Suite.SuiteClasses, it is currently not practical since we have many modules in our project and most of our tests comply with the module_name-->src-->test directory structure that maven requires to run tests. These tests are therefore not within @Suite.SuiteClasses, and maven automatically picks them and runs them at runtime.

My question is, how do I apply our RetryRunner on all our test classes that are not a part of @Suite.SuiteClasses? Since we have hundreds of test classes, going to each test class and prepending it with @RunWith(RetryRunner.class) would be our least preferred option since it is quite cumbersome.

Question1: Is there a way to pass RetryRunner as a maven command line arg in such a way that it will replace the normal Runner for all test classes?

Question2: Or is there a way to configure RetryRunner in the main project pom.xml so that it will be used on all test classes?

Swapnil
  • 1
  • Welcome Swapnil. I have not an answer to your question, but, honestly, unit-tests that are worth re-trying when failing seem like a test smell to me. Seems like your code has dependencies to non-deterministic stuff. Have you considered mocking these dependencies instead? – Dirk Herrmann Mar 16 '19 at 10:39
  • Hi Dirk, thanks for your comment. Actually these intermittent failures happen because occasionally, JERSEY duplicates our REST calls (specifically POST). Since the 1st REST call has already done the job, the 2nd REST call appears to fail and thus our whole test fails. At other times, tests fail because either the socket times out; OR because a component may be waiting for a resource for a longer than expected time. After a lot of deliberation, we have come to this decision that we want to retry tests that intermittently fail. The option of modifying tests has also been exhausted. – Swapnil Mar 18 '19 at 21:30
  • I see two alternatives from your answer: 1) Your intent is checking the whole chain, including libraries, sockets, timing etc. This would certainly be perfectly OK, but it would not be unit-testing but rather integration testing. If this is the case, please use the integration-testing tag instead of unit-testing: Otherwise your question will not be found by the proper experts. 2) Your intent is to unit-test your code. In this case, you should isolate it from non-deterministic stuff etc. If this is the case, search for mocking, inversion-of-control etc.. – Dirk Herrmann Mar 20 '19 at 22:17

0 Answers0