Specifying an order to junit 4 tests at the Method level (not class level)

Question

I know this is bad practice, but it needs to be done, or I'll need to switch to testng. Is there a way, similar to JUnit 3's testSuite, to specify the order of the tests to be run in a class?

Answer 1

I can see several reasons for doing this, especially when using JUnit to run functional tests or test persistent objects. For example, consider an object Article which is persisted to some kind of persistent storage. If I would like to test the insert, update and delete functionality on the Article object following the unit test principle "all tests should be reorderable and test only a specific part of the functionality", I would have three tests:

• testInsertArticle()
• testUpdateArticle()
• testDeleteArticle()

However, to be able to test the update functionality, I would first need to insert the article. To test the delete functionality, I would also need to insert an article. So, in practice, the insert functionality is already tested both in testUpdateArticle() and testDeleteArticle(). It is then tempting to just create a test method testArticleFunctionality() which does it all, but methods like that will eventually get huge (and they won't just test part of the functionality of the Article object).

The same goes for running functional tests against for example a restful API. JUnit is great also for these cases if it wasn't for the undeterministic ordering of tests.

That said, I extended Michael D's OrderedRunner to use annotations to determine order of tests, just thought I should share. It can be extended further, for example by specifying exactly which tests each test depends on, but this is what I'm using for now.

This is how it is used. It avoids the need for naming tests like AA_testInsert(), AB_testUpdate(), AC_testDelete(), ..., ZC_testFilter(), etc.

@RunWith(OrderedRunner.class)
public class SomethingTest {
    @Test
    @Order(order=2)
    public void testUpdateArticle() {
        // test update
    }

    @Test
    @Order(order=1)
    public void testInsertArticle() {
        // test insert
    }

    @Test
    @Order(order=3)
    public void testDeleteArticle() {
        // test delete
    }
}

No matter how these tests are placed in the file, they will always be run as order=1 first, order=2 second and last order=3, no matter if you run them from inside Eclipse, using Ant, or any other way.

Implementation follows. First, the annotation Order.

@Retention(RetentionPolicy.RUNTIME)
public @interface Order {
    public int order();
}

Then, the modified OrderedRunner.

public class OrderedRunner extends BlockJUnit4ClassRunner {
    public OrderedRunner(Class<?> klass) throws InitializationError {
        super(klass);
    }

    @Override
    protected List<FrameworkMethod> computeTestMethods() {
        List<FrameworkMethod> copy = new ArrayList<>(super.computeTestMethods());
        Collections.sort(list, new Comparator<FrameworkMethod>() {
            @Override
            public int compare(FrameworkMethod f1, FrameworkMethod f2) {
                Order o1 = f1.getAnnotation(Order.class);
                Order o2 = f2.getAnnotation(Order.class);
        
                if(o1==null && o2 == null) return 0;
                if (o1 == null) return 1;
                if (o2 == null) return -1;

                return o1.order() - o2.order();
            }
        });
        return list;
    }
}

Answer 2

If you're sure you really want to do this: There may be a better way, but this is all I could come up with...

JUnit4 has an annotation: @RunWith which lets you override the default Runner for your tests.

In your case you would want to create a special subclass of BlockJunit4ClassRunner, and override computeTestMethods() to return tests in the order you want them executed. For example, let's say I want to execute my tests in reverse alphabetical order:

public class OrderedRunner extends BlockJUnit4ClassRunner {

    public OrderedRunner(Class klass) throws InitializationError {
        super(klass);
    }

    @Override
    protected List computeTestMethods() {
        List list = super.computeTestMethods();
        List copy = new ArrayList(list);
        Collections.sort(copy, new Comparator() {
            public int compare(FrameworkMethod o1, FrameworkMethod o2) {
                return o2.getName().compareTo(o1.getName());
            }
        });
        return copy;
    }
}

@RunWith(OrderedRunner.class)
public class OrderOfTest {
    @Test public void testA() { System.out.println("A"); }
    @Test public void testC() { System.out.println("C"); }
    @Test public void testB() { System.out.println("B"); }
}

Running this test produces:

C
B
A

For your specific case, you would want a comparator that would sort the tests by name in the order you want them executed. (I would suggest defining the comparator using something like Google Guava's class Ordering.explicit("methodName1","methodName2").onResultOf(...); where onResultOf is provided a function that converts FrameworkMethod to its name... though obviously you are free to implement that any way you want.

Answer 3

From JUnit version 4.11 onwards, it is possible to influence the order of test execution by annotating your class with @FixMethodOrder and specifying any of the available MethodSorters. See this link for more details.

Answer 4

Using junit 4.11 the new annotation @FixMethodOrder allows to set a specific order:

@FixMethodOrder(MethodSorters.NAME_ASCENDING)

Answer 5

If you want to run junit tests in order "just as they present in your source code", and don't want to modify your tests code, see my note about this here:

How to run junit tests in order as they present in your source code

But it is really not a good idea, tests must be independent.

Answer 6

Joscarsson and Michael D code in my github repo. I hope they don't mind. I also provide ordered version for Parameterized class. It's already to use as maven dependency

<repositories>
    <repository>
        <id>git-xxx</id>
        <url>https://github.com/crsici/OrderedRunnerJunit4.11/raw/master/</url>
    </repository>
</repositories>

<dependency>
    <groupId>com.sici.org.junit</groupId>
    <artifactId>ordered-runner</artifactId>
    <version>0.0.1-RELEASE</version>
</dependency>