Is there a way to make Eclipse run a JUnit test multiple times until failure?

Question

We occasionally have bugs that appear once in every X runs. Before people check in stuff (where it is automatically JUnit'd), our devs need to pass JUnit locally via Eclipse.

Is there some convenient way (built in or high-quality Plugin) to make Eclipse run the same test X times and stop if there's a failure? An alternative to just clicking Run X times?

Note that I'm looking for something in the UI (e.g., right click and say "Run X times" instead of just "Run").

Answer 1

If the for loop works, then I agree with nos.

If you need to repeat the entire setup-test-teardown, then you can use a TestSuite:

1. Right-click on the package containing the test to repeat
2. Go to New and choose to create a JUnit test SUITE
3. Make sure that only the test you want to repeat is selected and click through to finish.
4. Edit the file to run it multiple times.

In the file you just find the

addTestSuite(YourTestClass.class)

line, and wrap that in a for loop.

I'm pretty sure that you can use addTest instead of addTestSuite to get it to only run one test from that class if you just want to repeat a single test method.

Answer 2

If you really want to run a test class until failure, you need your own runner.

@RunWith(RunUntilFailure.class)
public class YourClass {

    // ....

}

which could be implemented as follows...

package com.example;

import org.junit.internal.runners.*;
import org.junit.runner.notification.*;
import org.junit.runner.*;

public class RunUntilFailure extends Runner {

    private TestClassRunner runner;

    public RunUntilFailure(Class<?> klass) throws InitializationError {
        this.runner = new TestClassRunner(klass);
    }

    @Override
    public Description getDescription() {
        Description description = Description.createSuiteDescription("Run until failure");
        description.addChild(runner.getDescription());
        return description;
    }

    @Override
    public void run(RunNotifier notifier) {
        class L extends RunListener {
            boolean fail = false;
            public void testFailure(Failure failure) throws Exception { fail = true; }
        }
        L listener = new L();
        notifier.addListener(listener);
        while (!listener.fail) runner.run(notifier);
    }

}

...releasing untested code, feeling TDD guilt :)

Answer 3

Based on @akuhn's answer, here is what I came up with - rather than running forever, this will run 50 times or until failure, whichever comes first.

package com.foo    

import org.junit.runner.Description;
import org.junit.runner.Runner;
import org.junit.runner.notification.Failure;
import org.junit.runner.notification.RunListener;
import org.junit.runner.notification.RunNotifier;
import org.junit.runners.BlockJUnit4ClassRunner;
import org.junit.runners.model.InitializationError;

public class RunManyTimesUntilFailure extends Runner {

    private static final int MAX_RUN_COUNT = 50;

    private BlockJUnit4ClassRunner runner;

    @SuppressWarnings("unchecked")
    public RunManyTimesUntilFailure(final Class testClass) throws InitializationError {
        runner = new BlockJUnit4ClassRunner(testClass); 
    }

    @Override
    public Description getDescription() {
        final Description description = Description.createSuiteDescription("Run many times until failure");
        description.addChild(runner.getDescription());
        return description;
    }

    @Override
    public void run(final RunNotifier notifier) {
        class L extends RunListener {
            boolean shouldContinue = true;
            int runCount = 0;

            @Override
            public void testFailure(@SuppressWarnings("unused") final Failure failure) throws Exception {
                shouldContinue = false;
            }

            @Override
            public void testFinished(@SuppressWarnings("unused") Description description) throws Exception {
                runCount++;

                shouldContinue = (shouldContinue && runCount < MAX_RUN_COUNT);
            }
        }

        final L listener = new L();
        notifier.addListener(listener);

        while (listener.shouldContinue) {
            runner.run(notifier);
        }
    }
}

Answer 4

I know it doesn't answer the question directly but if a test isn't passing every time it is run it is a test smell known as Erratic Test. There are several possible causes for this (from xUnit Test Patterns):

• Interacting Tests
• Interacting Test Suites
• Lonely Test
• Resource Leakage
• Resource Optimism
• Unrepeatable Test
• Test Run War
• Nondeterministic Test

The details of each of these is documented in Chapter 16 of xUnit Test Patterns.

Answer 5

Here is a post I wrote that shows several ways of running the tests repeatedly with code examples: http://codehowtos.blogspot.com/2011/04/run-junit-test-repeatedly.html

You can use the @Parametrized runner, or use the special runner included in the post There is also a reference to a @Retry implementation

Answer 6

I don't believe there's a built in way for junit to do exactly what you're asking for.

If multiple runs produce different result, you should have a unit test testing that case. Wich might be as simple as running a for loop in the relevant test cases.

Answer 7

It is possible to implement such an loop with TestRules (since JUnit 4.9)

A very simple implementation that runs every Test 10 times:

import org.junit.rules.TestRule;
import org.junit.runner.Description;
import org.junit.runners.model.Statement;

public class SimpleRepeatRule implements TestRule {

    private static class SimpleRepeatStatement extends Statement {

        private final Statement statement;

        private SimpleRepeatStatement(Statement statement) {
            this.statement = statement;
        }

        @Override
        public void evaluate() throws Throwable {
            for (int i = 0; i < 10; i++) {
                statement.evaluate();
            }
        }
    }

    @Override
    public Statement apply(Statement statement, Description description) {
        return new SimpleRepeatStatement(statement);
    }
}

usage:

public class Run10TimesTest {

   @Rule
   public SimpleRepeatRule repeatRule = new SimpleRepeatRule();

   @Test
   public void myTest(){...}
}

For a more useful implementation based on an annotation that define which test method has to been executed how often have a look at this blog: http://www.codeaffine.com/2013/04/10/running-junit-tests-repeatedly-without-loops/