Repeating every test at most N times until first success

Question

The JUnit tests I have are annotated with

@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration(locations=...

The probability of each one of the tests, taken individually, to fail is quite low. However, due to the large number of tests, the probability that all of the tests succeed, thus passing the automatic tests, is also low. It is enough that the test succeeds once to know that the functionality tested was implemented correctly (when the tests fail, either they always fail because the functionality was not implemented correctly, or they fail occasionally, due to some combination of conditions, related to the testing environment, which rarely occurs).

So what I need is, for a fixed N, to repeat every test at most N times until success. Only if the test failed all N times, I deduce that something must be broken.

Can one suggest a clean way to implement this (subclassing the Spring JUnit runner, perhaps? some other way?)

Why do you have failing tests? It's usually a sign of bad design, or overly eager testing on things which shouldn't necessarily be tested on. Are they integration tests with components you do not have control over? — Koos Gadellaa, Jul 06 '16 at 19:14
@KoosGadellaa "Why do you have failing tests?" - it's a long story, and it is not possible to change the testing environment. The point is that I need only to know that the functionality is implemented correctly with sufficiently high probability, and this is a problem which needs to be solved quickly. — John Donn, Jul 06 '16 at 19:24
Maybe you can use [Contiperf](http://databene.org/contiperf) to run it N times, and then have a listener in each testcase which allows you to easily validate if a testcase has a succesful conclusion? Doesn't stop after a first successful run, however — Koos Gadellaa, Jul 06 '16 at 19:39
You are violating the principle of reliable, reproducible tests. If your testing environment is unstable (databases up and down, unreliable third party) you should consider mocking them. Tests should run and suceed. Period — vikingsteve, Jul 07 '16 at 12:41
@vikingsteve basically I agree. However mocking external components violates the purpose of integration testing, and even if I wanted to do that, in my case things are more complicated and it is not really possible. — John Donn, Jul 07 '16 at 13:50
I would argue then that the mechanism to repeat an action N times should be built into your code, and perhaps configurable via `http.retry.count=3` or whatever. Your integration test should still pass every time. If there's still an issue with dependent systems you don't have control over or third parties you address it with them. — vikingsteve, Jul 07 '16 at 15:43
@vikingsteve "..should be built into your code.." - if I understand correctly, sorry if I don't, you suggest to insert the **for (int i=0; i — John Donn, Jul 07 '16 at 16:08
No I'm not suggesting to put that iteration loop in your test methods, I'm suggesting if you really need that sort of "resilience" due to instability in database or third party systems or whatever then your application should behave that way. The tests should just run once and either suceed or fail. At least that's how I view it. Since if you have to run a test 3 times to suceed - how will your system have resilience in production environments?? — vikingsteve, Jul 07 '16 at 18:32
@vikingsteve thank you, however in my case the (rare non systematic) errors are specific to the testing environment. — John Donn, Jul 08 '16 at 06:23
Does this help http://stackoverflow.com/questions/1492856/easy-way-of-running-the-same-junit-test-over-and-over — vikingsteve, Jul 08 '16 at 08:30

John Donn · Answer 1 · 2016-07-11T19:15:59.353

Here is a basic implementation which seems to work (one annotates the test classes for which the test methods are to be repeated until success with @CustomJUnit4ClassRunner instead of @SpringJUnit4ClassRunner ):

import org.junit.internal.runners.model.ReflectiveCallable;
import org.junit.internal.runners.statements.Fail;
import org.junit.runners.model.FrameworkMethod;
import org.junit.runners.model.InitializationError;
import org.junit.runners.model.Statement;
import org.springframework.test.context.junit4.SpringJUnit4ClassRunner;

public class CustomJUnit4ClassRunner extends SpringJUnit4ClassRunner {

    public CustomJUnit4ClassRunner(Class<?> clazz)
            throws InitializationError {
        super(clazz);
    }

    @Override
    protected Statement methodBlock(FrameworkMethod frameworkMethod) {
        Object testInstance;
        try {
            testInstance = new ReflectiveCallable() {

                @Override
                protected Object runReflectiveCall() throws Throwable {
                    return createTest();
                }
            }.run();
        }
        catch (Throwable ex) {
            return new Fail(ex);
        }

        Statement statement = methodInvoker(frameworkMethod, testInstance);
        statement = possiblyExpectingExceptions(frameworkMethod, testInstance, statement);
        statement = withBefores(frameworkMethod, testInstance, statement);
        statement = withAfters(frameworkMethod, testInstance, statement);
        //statement = withRulesReflectively(frameworkMethod, testInstance, statement);
        //statement = withPotentialRepeat(frameworkMethod, testInstance, statement);
        statement = withPotentialTimeout(frameworkMethod, testInstance, statement);
        statement = withRepeatUntilSuccess(frameworkMethod, testInstance, statement);

        return statement;
    }

    private Statement withRepeatUntilSuccess(FrameworkMethod frameworkMethod, Object testInstance, Statement next) {
        return new CustomTestRepeat(next, frameworkMethod.getMethod(), Constants.MAX_NUM);
    }

}

together with the following class

import java.lang.reflect.Method;

import org.junit.runners.model.Statement;

public class CustomTestRepeat extends Statement {

    private final Statement next;

    private final Method testMethod;

    private final int repeat;

    public CustomTestRepeat(Statement next, Method testMethod, int repeat) {
        this.next = next;
        this.testMethod = testMethod;
        this.repeat = Math.max(1, repeat);
    }

    @Override
    public void evaluate() throws Throwable {
        for (int i = 0; i < this.repeat - 1; i++) {
            try {
                this.next.evaluate();
            }
            //Assertion errors are not instances of java.lang.Exception
            catch (Throwable e) {
                continue;
            }
            return;
        }
        this.next.evaluate();
    }
}

Repeating every test at most N times until first success

1 Answers1