10

A newbie to JUnit (in fact JUnit 4) and came across suite way of executing test

@RunWith(Suite.class)
@Suite.SuiteClasses(
        {                               
                CreateNewProfile.class,
                EditProfile.class,

        })
public class ProfileTestSuite {

}

This is the code sample I came across while browsing through the test code base at my new employer. During execution I fund that - CreateNewProfile tests are executed first and then EditProfile, which does make sense but then it introduces dependency among tests.

I have been following non dependent test mechanism from couple of months (though I used to use TestNG and not JUnit) and would expect EditProfile being able to be executed in isolation as well. That is edit profile should take care of creating profile and then editing it and then asserting the operations.

My question here is - has Junit 4 introduced test ordering feature. Is this feature intended or one easter egg as I always felt JUnit = independent tests.

Matthew Farwell
  • 60,889
  • 18
  • 128
  • 171
Tarun
  • 3,456
  • 10
  • 48
  • 82

6 Answers6

21

No JUnit does not support test ordering, except in the manner that you say, through Suite. This only defines the order in which the test classes are executed. This has been there for a long time, including JUnit 3 Suite classes.

For a more complete explanation, there are three things we need to talk about here:

  1. the ordering of tests classes within a test suite
  2. the ordering of test classes when they are found by reflection by Eclipse or Maven
  3. the ordering of test methods (annotated with @Test) within a test class.

The ordering of tests classes within a test suite

When you specify the list of classes to be executed in a test suite, you're defining an array, and these test classes will be executed in order, except when you're doing parallel execution. Unfortunately, this allows the introduction of dependencies between your test classes.

The ordering of test classes when they are found by reflection

When searching the classpath for classes, the order in which they are found is not guaranteed, so cannot be depended upon. It isn't actually JUnit which is doing the searching, but the Eclipse Junit plugin, or maven surefire or failsafe.

The ordering of test methods within a test class

JUnit does not guarantee the order of execution of tests within a class. Most of the time, on most JVMs pre version 7, the order in which they are found using reflection is in declaration order, ie the order they are in the file. This is the order in which they are executed. However, with JVM 7, this is no longer guaranteed, so there won't be a consistent order. There is a github issue #293 Sort test methods for predictability open with suggested solutions, and there is a thread on the junit mailing list: Alphabetizing test method run order?. So you cannot depend upon the order that tests will be executed using JUnit, but this is currently under discussion.

Matthew Farwell
  • 60,889
  • 18
  • 128
  • 171
  • So I take from here that test should be able to be executed independently and not depending on the order they are arranged in @Suite. And this is going to give me surprises when I execute tests in parallel. Right? – Tarun Oct 21 '11 at 09:33
  • Test methods (and therefore classes) should always be executable independently, if only so that you can execute a single test in an IDE. It depends how your parallel runner executes the tests, does it execute the classes in parallel or the methods within the classes in parallel, but yes it can cause problems. It's better to try and avoid all dependencies between tests. – Matthew Farwell Oct 21 '11 at 09:46
  • 2
    Matthew: only unit tests should be independent. For functional testing, being able to order methods specifically is very useful, which is why it's been a feature in TestNG since version 1. – Cedric Beust Oct 21 '11 at 16:29
5

No, ordering is not supported. And yes, it is intention:

KentBeck commented (December 30, 2010): Independent tests are more valuable, so JUnit by design does not support test ordering.

If you need this feature, you could use TestNG with it's @Test(sequential = true).

jeha
  • 10,562
  • 5
  • 50
  • 69
  • Well, every time I execute it I see methods of CreateNewProfile.class are invoked first and then EditProfile.class. Now is not it ordering? – Tarun Oct 21 '11 at 07:06
  • @Tarun: I guess this is the result of how it is implemented (implicit ordering). The point is that a sequence of execution is not guaranteed (no explicit ordering), so this might change and you can't rely on this. – jeha Oct 21 '11 at 07:11
  • Using \@Suite to order Test Classes worries me as I can not execute them independently when I execute them out of \@suite – Tarun Oct 21 '11 at 07:18
  • Actually, with TestNG, it's @Test(dependsOnMethods=...) or, recommended, @Test(dependsOnGroups=...). – Cedric Beust Oct 21 '11 at 16:29
  • 1
    Ordering can be achieved in JUnit with the class annotation ``@FixMethodOrder(MethodSorters.NAME_ASCENDING)`` (since JUnit 4.11) but it should be avoided for the already mentioned reasons. – nrainer Nov 06 '14 at 14:10
3

@jeha answer is very good but he talks about test methods (the ones annotated with @Test) order. This order is indeed unspecified, probably because the order of methods in a class obtained via reflection is not guaranteed to correspond with the order of methods in the source file.

However from my experience they are executed in the same order they were defined most of the time.

You are asking about the order of test classes. This should not be a surprise: you are explicitly listing test classes using @Suite.SuiteClasses and JUnit probably has no business in shuffling them and running in different order. Are you concerned about introducing dependency between tests simply because you explicitly ordered them?

Tomasz Nurkiewicz
  • 334,321
  • 69
  • 703
  • 674
  • Yup I am worried about the ordering which is introduced in execution of test classes specified in \@Suite. I feel it's misuse of \@Suite and I can not just execute "EditProfile.class" out of @Suite if I wanted to. – Tarun Oct 21 '11 at 07:16
1

You can extend the Suite runner:

public class MySuite extends Suite {

    public SuiteRunner(Class<?> klass, RunnerBuilder builder) throws InitializationError {
        super(klass, builder);
    }

    @Override
    protected List<Runner> getChildren() {
        List<Runner> children = super.getChildren();
        ... here modify the children list - you can remove or reorder test...
        return children;
    }
}

Then annotate your suite:

@RunWith(MySuite.class)
@Suite.SuiteClasses({                               
        CreateNewProfile.class,
        EditProfile.class})
public class ProfileTestSuite { }

The tests marked with @Ignore will be represented by IgnoredClassRunner and regular test will be represented by BlockJUnit4ClassRunner (or whatever runner you will annotate them - which will be probably extension of BlockJUnit4ClassRunner).

From the runner you can get the test class. Then you can use own annotations.

Masáč
  • 163
  • 1
  • 5
1

JUnit 4.11 now supports specifying execution order using @FixMethodOrder annoation.

Aleksandr Dubinsky
  • 22,436
  • 15
  • 82
  • 99
0

Just bypass it by ordering the names of the class by using an _# before the class name. Ex _1testcaselogin _2testcaseverify

Kelly Ryan
  • 1