How to Re-run failed JUnit tests immediately?

Question

Is there a way to have an JUnit Rule or something similar that gives every failing test a second chance, just by trying to run it once again.

Background: I have a large Set of Selenium2-WebDriver tests written with JUnit. Due to a very aggressive timing (only short wait periods after the clicks) some tests (1 out of 100, and always a different one) can fail because the server sometimes responds a bit slower. But I can not make the wait period so long that it is definitely long enough, because then the tests will take for ever.) -- So I think it is acceptable for this use case that a test is green even if it needs a second try.

Of course it would be better to have a 2 out of 3 majority (repeat a failing test 3 times, and take them as correct, if two of the tests are correct), but this would be a future improvement.

Answer 1

You can do this with a TestRule. This will give you the flexibility you need. A TestRule allows you to insert logic around the test, so you would implement the retry loop:

public class RetryTest {
    public class Retry implements TestRule {
        private int retryCount;

        public Retry(int retryCount) {
            this.retryCount = retryCount;
        }

        public Statement apply(Statement base, Description description) {
            return statement(base, description);
        }

        private Statement statement(final Statement base, final Description description) {
            return new Statement() {
                @Override
                public void evaluate() throws Throwable {
                    Throwable caughtThrowable = null;

                    // implement retry logic here
                    for (int i = 0; i < retryCount; i++) {
                        try {
                            base.evaluate();
                            return;
                        } catch (Throwable t) {
                            caughtThrowable = t;
                            System.err.println(description.getDisplayName() + ": run " + (i+1) + " failed");
                        }
                    }
                    System.err.println(description.getDisplayName() + ": giving up after " + retryCount + " failures");
                    throw caughtThrowable;
                }
            };
        }
    }

    @Rule
    public Retry retry = new Retry(3);

    @Test
    public void test1() {
    }

    @Test
    public void test2() {
        Object o = null;
        o.equals("foo");
    }
}

The heart of a TestRule is the base.evaluate(), which calls your test method. So around this call you put a retry loop. If an exception is thrown in your test method (an assertion failure is actually an AssertionError), then the test has failed, and you'll retry.

There is one other thing that may be of use. You may only want to apply this retry logic to a set of tests, in which case you can add into the Retry class above a test for a particular annotation on the method. Description contains a list of annotations for the method. For more information about this, see my answer to How to run some code before each JUnit @Test method individually, without using @RunWith nor AOP?.

Using a custom TestRunner

This is the suggestion of CKuck, you can define your own Runner. You need to extend BlockJUnit4ClassRunner and override runChild(). For more information see my answer to How to define JUnit method rule in a suite?. This answer details how to define how to run code for every method in a Suite, for which you have to define your own Runner.

Answer 2

As for me writing custom runner more flexible solution. The solution that posted above (with code example) has two disadvantages:

1. It won't retry test if it fails on the @BeforeClass stage;
2. It calculating tests run a bit differently (when you have 3 retries, you will receive test Runs: 4, success 1 that might be confusing);

That's why I prefer more approach with writing custom runner. And code of custom runner could be following:

import org.junit.Ignore;
import org.junit.internal.AssumptionViolatedException;
import org.junit.internal.runners.model.EachTestNotifier;
import org.junit.runner.Description;
import org.junit.runner.notification.RunNotifier;
import org.junit.runner.notification.StoppedByUserException;
import org.junit.runners.BlockJUnit4ClassRunner;
import org.junit.runners.model.FrameworkMethod;
import org.junit.runners.model.InitializationError;
import org.junit.runners.model.Statement;


public class RetryRunner extends BlockJUnit4ClassRunner {

    private final int retryCount = 100;
    private int failedAttempts = 0;

    public RetryRunner(Class<?> klass) throws InitializationError {
        super(klass);
    }    


    @Override
    public void run(final RunNotifier notifier) {
        EachTestNotifier testNotifier = new EachTestNotifier(notifier,
                getDescription());
        Statement statement = classBlock(notifier);
        try {

            statement.evaluate();
        } catch (AssumptionViolatedException e) {
            testNotifier.fireTestIgnored();
        } catch (StoppedByUserException e) {
            throw e;
        } catch (Throwable e) {
            retry(testNotifier, statement, e);
        }
    }

    @Override
    protected void runChild(final FrameworkMethod method, RunNotifier notifier) {
        Description description = describeChild(method);
        if (method.getAnnotation(Ignore.class) != null) {
            notifier.fireTestIgnored(description);
        } else {
            runTestUnit(methodBlock(method), description, notifier);
        }
    }

    /**
     * Runs a {@link Statement} that represents a leaf (aka atomic) test.
     */
    protected final void runTestUnit(Statement statement, Description description,
            RunNotifier notifier) {
        EachTestNotifier eachNotifier = new EachTestNotifier(notifier, description);
        eachNotifier.fireTestStarted();
        try {
            statement.evaluate();
        } catch (AssumptionViolatedException e) {
            eachNotifier.addFailedAssumption(e);
        } catch (Throwable e) {
            retry(eachNotifier, statement, e);
        } finally {
            eachNotifier.fireTestFinished();
        }
    }

    public void retry(EachTestNotifier notifier, Statement statement, Throwable currentThrowable) {
        Throwable caughtThrowable = currentThrowable;
        while (retryCount > failedAttempts) {
            try {
                statement.evaluate();
                return;
            } catch (Throwable t) {
                failedAttempts++;
                caughtThrowable = t;
            }
        }
        notifier.addFailure(caughtThrowable);
    }
}

Answer 3

Now there is a better option. If you're using maven plugins like: surfire or failsefe there is an option to add parameter rerunFailingTestsCount SurFire Api. This stuff was implemented in the following ticket: Jira Ticket. In this case you don't need to write your custom code and plugin automatically amend test results report.
I see only one drawback of this approach: If some test is failed on Before/After class stage test won't be re-ran.

Answer 4

Proposed comment was written based ob this article with some additions.

Here, if some Test Case from your jUnit project gets "failure" or "error" result, this Test Case will be re-run one more time. Totally here we set 3 chance to get success result.

So, We need to create Rule Class and add "@Rule" notifications to your Test Class.

If you do not want to wright the same "@Rule" notifications for each your Test Class, you can add it to your abstract SetProperty Class(if you have it) and extends from it.

Rule Class:

import org.junit.rules.TestRule;
import org.junit.runner.Description;
import org.junit.runners.model.Statement;

public class RetryRule implements TestRule {
    private int retryCount;

    public RetryRule (int retryCount) {
        this.retryCount = retryCount;
    }

    public Statement apply(Statement base, Description description) {
        return statement(base, description);
    }

    private Statement statement(final Statement base, final Description description) {
        return new Statement() {
            @Override
            public void evaluate() throws Throwable {
                Throwable caughtThrowable = null;

                // implement retry logic here
                for (int i = 0; i < retryCount; i++) {
                    try {
                        base.evaluate();
                        return;
                    } catch (Throwable t) {
                        caughtThrowable = t;
                        //  System.out.println(": run " + (i+1) + " failed");
                        System.err.println(description.getDisplayName() + ": run " + (i + 1) + " failed.");
                    }
                }
                System.err.println(description.getDisplayName() + ": giving up after " + retryCount + " failures.");
                throw caughtThrowable;
            }
        };
    }
}

Test Class:

import org.junit.BeforeClass;
import org.junit.Rule;
import org.junit.Test;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.firefox.FirefoxDriver;
import static org.hamcrest.CoreMatchers.is;
import static org.hamcrest.MatcherAssert.assertThat;

/**
 * Created by ONUR BASKIRT on 27.03.2016.
 */
public class RetryRuleTest {

    static WebDriver driver;
    final private String URL = "http://www.swtestacademy.com";

    @BeforeClass
    public static void setupTest(){
        driver = new FirefoxDriver();
    }

    //Add this notification to your Test Class 
    @Rule
    public RetryRule retryRule = new RetryRule(3);

    @Test
    public void getURLExample() {
        //Go to www.swtestacademy.com
        driver.get(URL);

        //Check title is correct
        assertThat(driver.getTitle(), is("WRONG TITLE"));
    }
}

Answer 5

You have to write your own org.junit.runner.Runner and annotate your tests with @RunWith(YourRunner.class).

Answer 6

As for Junit5 there is a cool feature @RetryingTest offered by junit-pioneer extension https://junit-pioneer.org/docs/retrying-test/, simple example:

public class RetryTest {
    private int counter = 0;


    @RetryingTest(5)
    void retryCounter() {
        if (counter++ < 2) fail();
    }

}

This will fail twice and show green on third execution.

Answer 7

This answer is built on this answer.

If you need your ActivityScenario (and your Activity) to be recreated before each run, you can launch it using try-with-resources. The ActivityScenario will then be closed automatically after each try.

public final class RetryRule<A extends Activity> implements TestRule {
    private final int retryCount;
    private final Class<A> activityClazz;
    private ActivityScenario<A> scenario;

    /**
     * @param retryCount the number of retries. retryCount = 1 means 1 (normal) try and then
     * 1 retry, i.e. 2 tries overall
     */
    public RetryRule(int retryCount, @NonNull Class<A> clazz) {
        this.retryCount = retryCount;
        this.activityClazz = clazz;
    }

    public Statement apply(Statement base, Description description) {
        return statement(base, description);
    }

    private Statement statement(final Statement base, final Description description) {
        return new Statement() {
            @Override
            public void evaluate() throws Throwable {
                Throwable caughtThrowable = null;

                // implement retry logic here
                for (int i = 0; i <= retryCount; i++) {
                    try(ActivityScenario<A> scenario = ActivityScenario.launch(activityClazz)){
                        RetryRule.this.scenario = scenario;
                        base.evaluate();
                        return;
                    } catch (Throwable t) {
                        caughtThrowable = t;
                        Log.e(LOGTAG,
                                description.getDisplayName() + ": run " + (i + 1) + " failed: ", t);
                    }
                }
                Log.e(LOGTAG,
                        description.getDisplayName() + ": giving up after " + (retryCount + 1) +
                                " failures");
                throw Objects.requireNonNull(caughtThrowable);
            }
        };
    }

    public ActivityScenario<A> getScenario() {
        return scenario;
    }
}

You can then access your scenario in your tests using the getScenario() method.