How do I define an order to run junit tests in Intellij?

Question

I have a flaky junit test that only fails if I run all my tests. I think that one test is causing another test to fail, I want to prove it before I try to fix it.

If I run all tests, it runs the "bad setup" then it runs the "test that fails after bad setup". It also runs a lot of irrelevant, slow tests in between. But if I use a pattern to only run these two, it runs "test that fails after bad setup" then "bad setup". As a result, both pass.

How do I only run "bad setup" and "test that fails after bad setup", in that order?

@Lashane I need to *know* the problem to fix the test. This question is about learning what the problem is. — Daniel Kaplan, Dec 24 '15 at 18:45
usually "debugging" is the way to solve it, run all tests, find where it fails, put breakpoint before - find out why it actually fails, what state/variable lead to it, find where this state is changed, put breakpoints and run all tests again — Iłya Bursov, Dec 24 '15 at 18:48
@Lashane I do that in general, but in this case it's an extremely inefficient way to debug this issue. — Daniel Kaplan, Dec 25 '15 at 01:52

Ali Dehghani · Answer 1 · 2017-07-05T19:02:26.803

According to JUnit's wiki:

By design, JUnit does not specify the execution order of test method invocations. Until now, the methods were simply invoked in the order returned by the reflection API. However, using the JVM order is unwise since the Java platform does not specify any particular order, and in fact JDK 7 returns a more or less random order. Of course, well-written test code would not assume any order, but some do, and a predictable failure is better than a random failure on certain platforms.

From version 4.11, JUnit will by default use a deterministic, but not predictable, order (MethodSorters.DEFAULT). To change the test execution order simply annotate your test class using @FixMethodOrder and specify one of the available MethodSorters:

@FixMethodOrder(MethodSorters.JVM): Leaves the test methods in the order returned by the JVM. This order may vary from run to run.

@FixMethodOrder(MethodSorters.NAME_ASCENDING): Sorts the test methods by method name, in lexicographic order.

You could use MethodSorters.NAME_ASCENDING and change your method names to match with your specific order. I know you're using this just for debugging sake but it's a Test Smell to rely on your test methods execution order and JUnit does not provide more finer grain control over test methods execution order

Cool, that's helpful. In the end, I just manually put @Ignore on everything (temporarily) — Daniel Kaplan, Dec 24 '15 at 23:00
This doesn't help this scenario because this problem spans multiple classes. — Daniel Kaplan, Mar 30 '17 at 16:13

score 7 · Answer 2 · answered Dec 17 '16 at 12:40

As said by Ali Dehghani, You can order the test method execution by

@FixMethodOrder(MethodSorters.NAME_ASCENDING): Sorts the test methods by method name, in lexicographic order.

Code:

@FixMethodOrder(MethodSorters.NAME_ASCENDING)
public class ApplicationTest extends ActivityInstrumentationTestCase2<MainActivity> {

    public ApplicationTest() {
        super(MainActivity.class);
    }

    @Rule
    public ActivityTestRule<MainActivity> mActivityTestRule = new ActivityTestRule<>(MainActivity.class);

    @Test
    void t1AttachUI(){
        // testing code goes here
    }

    @Test
    void t2InitializeViews(){
        // testing code goes here
    };

    @Test
    void t3SettingValues(){
        // testing code goes here
    };

    @Test
    void t4Validation(){
        // testing code goes here
    };

    @Test
    void t3AfterButtonPress(){
        // testing code goes here
    };
}

This doesn't help this scenario because this problem spans multiple classes. — Daniel Kaplan, Dec 20 '16 at 18:13
Naming t1, t2, ... working only up to t9. Naming t01, t02, ... working up to t99. — t0m, Oct 28 '19 at 14:21

score 2 · Answer 3 · answered Dec 24 '15 at 18:51

Unit tests ought to be independent so most frameworks don't guarantee or enforce the order in which they are run. But since you want to enforce an order the easiest way I've done it in the past it to create a "throw away" test suite or test method that calls the tests in whatever order I want them to run in. Unit tests are methods, just call them. This is easy to do if you're dealing with tens of tests, not at all appealing if you're dealing with hundreds or thousands.

Try to isolate the flaky interaction as much as possible, then swap around the order of the poorly interacting tests within the throwaway calling method.

How do I define an order to run junit tests in Intellij?

3 Answers3

Linked