Easy way of running the same junit test over and over?

Question

Like the title says, I'm looking for some simple way to run JUnit 4.x tests several times in a row automatically using Eclipse.

An example would be running the same test 10 times in a row and reporting back the result.

We already have a complex way of doing this but I'm looking for a simple way of doing it so that I can be sorta sure that the flaky test I've been trying to fix stays fixed.

An ideal solution would be an Eclipse plugin/setting/feature that I am unaware of.

Answer 1

The easiest (as in least amount of new code required) way to do this is to run the test as a parametrized test (annotate with an @RunWith(Parameterized.class) and add a method to provide 10 empty parameters). That way the framework will run the test 10 times.

This test would need to be the only test in the class, or better put all test methods should need to be run 10 times in the class.

Here is an example:

@RunWith(Parameterized.class)
public class RunTenTimes {

    @Parameterized.Parameters
    public static Object[][] data() {
        return new Object[10][0];
    }

    public RunTenTimes() {
    }

    @Test
    public void runsTenTimes() {
        System.out.println("run");
    }
}

With the above, it is possible to even do it with a parameter-less constructor, but I'm not sure if the framework authors intended that, or if that will break in the future.

If you are implementing your own runner, then you could have the runner run the test 10 times. If you are using a third party runner, then with 4.7, you can use the new @Rule annotation and implement the MethodRule interface so that it takes the statement and executes it 10 times in a for loop. The current disadvantage of this approach is that @Before and @After get run only once. This will likely change in the next version of JUnit (the @Before will run after the @Rule), but regardless you will be acting on the same instance of the object (something that isn't true of the Parameterized runner). This assumes that whatever runner you are running the class with correctly recognizes the @Rule annotations. That is only the case if it is delegating to the JUnit runners.

If you are running with a custom runner that does not recognize the @Rule annotation, then you are really stuck with having to write your own runner that delegates appropriately to that Runner and runs it 10 times.

Note that there are other ways to potentially solve this (such as the Theories runner) but they all require a runner. Unfortunately JUnit does not currently support layers of runners. That is a runner that chains other runners.

Answer 2

With IntelliJ, you can do this from the test configuration. Once you open this window, you can choose to run the test any number of times you want,.

when you run the test, intellij will execute all tests you have selected for the number of times you specified.

Example running 624 tests 10 times:

Answer 3

With JUnit 5 I was able to solve this using the @RepeatedTest annotation:

@RepeatedTest(10)
public void testMyCode() {
    //your test code goes here
}

Note that @Test annotation shouldn't be used along with @RepeatedTest.

Answer 4

I've found that Spring's repeat annotation is useful for that kind of thing:

@Repeat(value = 10)

Latest (Spring Framework 4.3.11.RELEASE API) doc:

• org.springframework.test.annotation
• Unit Testing in Spring

Answer 5

Inspired by the following resources:

• blog post
• this solution
• commented version

Example

Create and use a @Repeat annotation as follows:

public class MyTestClass {

    @Rule
    public RepeatRule repeatRule = new RepeatRule();

    @Test
    @Repeat(10)
    public void testMyCode() {
        //your test code goes here
    }
}

Repeat.java

import static java.lang.annotation.ElementType.ANNOTATION_TYPE;
import static java.lang.annotation.ElementType.METHOD;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

@Retention( RetentionPolicy.RUNTIME )
@Target({ METHOD, ANNOTATION_TYPE })
public @interface Repeat {
    int value() default 1;
}

RepeatRule.java

import org.junit.rules.TestRule;
import org.junit.runner.Description;
import org.junit.runners.model.Statement;

public class RepeatRule implements TestRule {

    private static class RepeatStatement extends Statement {
        private final Statement statement;
        private final int repeat;    

        public RepeatStatement(Statement statement, int repeat) {
            this.statement = statement;
            this.repeat = repeat;
        }

        @Override
        public void evaluate() throws Throwable {
            for (int i = 0; i < repeat; i++) {
                statement.evaluate();
            }
        }

    }

    @Override
    public Statement apply(Statement statement, Description description) {
        Statement result = statement;
        Repeat repeat = description.getAnnotation(Repeat.class);
        if (repeat != null) {
            int times = repeat.value();
            result = new RepeatStatement(statement, times);
        }
        return result;
    }
}

PowerMock

Using this solution with @RunWith(PowerMockRunner.class), requires updating to Powermock 1.6.5 (which includes a patch).

Answer 6

Anything wrong with:

@Test
void itWorks() {
    // stuff
}

@Test
void itWorksRepeatably() {
    for (int i = 0; i < 10; i++) {
        itWorks();
    }
}

Unlike the case where you are testing each of an array of values, you don't particularly care which run failed.

No need to do in configuration or annotation what you can do in code.

Answer 7

This works much easier for me.

public class RepeatTests extends TestCase {

    public static Test suite() {
        TestSuite suite = new TestSuite(RepeatTests.class.getName());

        for (int i = 0; i < 10; i++) {              
        suite.addTestSuite(YourTest.class);             
        }

        return suite;
    }
}

Answer 8

There's an Intermittent annotation in the tempus-fugit library which works with JUnit 4.7's @Rule to repeat a test several times or with @RunWith.

For example,

@RunWith(IntermittentTestRunner.class)
public class IntermittentTestRunnerTest {

   private static int testCounter = 0;

   @Test
   @Intermittent(repition = 99)
   public void annotatedTest() {
      testCounter++;
   }
}

After the test is run (with the IntermittentTestRunner in the @RunWith), testCounter would be equal to 99.

Answer 9

This is essentially the answer that Yishai provided above, re-written in Kotlin :

@RunWith(Parameterized::class)
class MyTest {

    companion object {

        private const val numberOfTests = 200

        @JvmStatic
        @Parameterized.Parameters
        fun data(): Array<Array<Any?>> = Array(numberOfTests) { arrayOfNulls<Any?>(0) }
    }

    @Test
    fun testSomething() { }
}

Answer 10

I build a module that allows do this kind of tests. But it is focused not only in repeat. But in guarantee that some piece of code is Thread safe.

https://github.com/anderson-marques/concurrent-testing

Maven dependency:

<dependency>
    <groupId>org.lite</groupId>
    <artifactId>concurrent-testing</artifactId>
    <version>1.0.0</version>
</dependency>

Example of use:

package org.lite.concurrent.testing;

import org.junit.Assert;
import org.junit.Rule;
import org.junit.Test;
import ConcurrentTest;
import ConcurrentTestsRule;

/**
 * Concurrent tests examples
 */
public class ExampleTest {

    /**
     * Create a new TestRule that will be applied to all tests
     */
    @Rule
    public ConcurrentTestsRule ct = ConcurrentTestsRule.silentTests();

    /**
     * Tests using 10 threads and make 20 requests. This means until 10 simultaneous requests.
     */
    @Test
    @ConcurrentTest(requests = 20, threads = 10)
    public void testConcurrentExecutionSuccess(){
        Assert.assertTrue(true);
    }

    /**
     * Tests using 10 threads and make 20 requests. This means until 10 simultaneous requests.
     */
    @Test
    @ConcurrentTest(requests = 200, threads = 10, timeoutMillis = 100)
    public void testConcurrentExecutionSuccessWaitOnly100Millissecond(){
    }

    @Test(expected = RuntimeException.class)
    @ConcurrentTest(requests = 3)
    public void testConcurrentExecutionFail(){
        throw new RuntimeException("Fail");
    }
}

This is a open source project. Feel free to improve.

Answer 11

You could run your JUnit test from a main method and repeat it so many times you need:

package tests;

import static org.junit.Assert.*;

import org.junit.Test;
import org.junit.runner.Result;

public class RepeatedTest {

    @Test
    public void test() {
        fail("Not yet implemented");
    }

    public static void main(String args[]) {

        boolean runForever = true;

        while (runForever) {
            Result result = org.junit.runner.JUnitCore.runClasses(RepeatedTest.class);

            if (result.getFailureCount() > 0) {
                runForever = false;
               //Do something with the result object

            }
        }

    }

}