Stopping Test's if JUnit Test fails

Question

I'm trying to figure out How i can make my JUnits tests run a specific order as well as stop all test's if one of the test's fail. Here below i have included just some simple actionbar item test's. All i basically want to make sure is when i run my build/test via command line, i want the test to stop if one of these tests fails. Any Recommendations?

public void testPreconditions() {
    assertNotNull(instrumentation);
    assertNotNull(mWeb);
    assertNotNull(mActivity);
}

protected void setUp() throws Exception {
    super.setUp();

    instrumentation = getInstrumentation();
    mActivity = getActivity();
    setActivityInitialTouchMode(false);
    mWeb = (WebView) mActivity
            .findViewById(com.jaisonbrooks.webview.R.id.webview_main);
    mMockWebViewClient = new MockWebViewClient();
    mWeb.setWebViewClient(mMockWebViewClient);
    mSettings = mWeb.getSettings();
}

public void testThatButtonReloadWorks() {
    final View view = mActivity
            .findViewById(com.jaisonbrooks.webview.R.id.menu_refresh);
    mActivity.runOnUiThread(new Runnable() {

        @Override
        public void run() {
            view.requestFocus();
            view.callOnClick();
        }
    });
}

public void testThatButtonForwardWorks() {
    final View view = mActivity
            .findViewById(com.jaisonbrooks.webview.R.id.menu_forward);
    mActivity.runOnUiThread(new Runnable() {

        @Override
        public void run() {
            view.requestFocus();
            view.callOnClick();
        }
    });
}

public void testThatButtonBackWorks() {
    final View view = mActivity
            .findViewById(com.jaisonbrooks.webview.R.id.menu_back);
    mActivity.runOnUiThread(new Runnable() {

        @Override
        public void run() {
            view.requestFocus();
            view.callOnClick();
        }
    });
}

You're breaking one of the fundamental principles of Junit - that all tests should be independent of each other. A good test class should have tests which can be executed in any order and don't have dependencies between them. — planetjones, Mar 18 '13 at 15:53
also see http://stackoverflow.com/questions/3693626/how-to-run-test-methods-in-spec-order-in-junit4 — planetjones, Mar 18 '13 at 15:55
There is a [pretty nifty example from IBM DeveloperWorks](https://www.ibm.com/developerworks/mydeveloperworks/blogs/amcohen/entry/stop_junit_before_you_get_into_trouble3?lang=en) of using a custom suite to accomplish this. You could also run your tests with Ant, which has an attribute on the Junit task that will allow you to halt on failure. — Perception, Mar 18 '13 at 15:56
@Perception Thanks for this, do you happen to know what that attribute is called possibly? — Jaison Brooks, Mar 18 '13 at 16:01
On the [Ant JUnit](http://ant.apache.org/manual/Tasks/junit.html) task the attribute you would be looking for is called `haltonfailure`. — Perception, Mar 18 '13 at 16:04

score 1 · Accepted Answer · edited May 23 '17 at 12:01

If you're doing integration tests using JUnit, you could prefix the method name with _1, _2 etc.:

public void _1_testThatButton()...
public void _2_testSomethingSecond()...

I suppose you could call System.exit() instead of throwing an exception, but if your tests are taking too long, that may be a bad sign for adding more tests in the future (adding more tests may discourage you from running them).

If your're using ant, you can try this: How to make junit testing to stop after first failing test

score 1 · Answer 2 · answered Mar 18 '13 at 15:56

I Haven't done Junit test for Android Apps, but based on my experience with web application, I am suggesting this answer.

When ever I face such situation, I'll call them in separate method with return type

Example:

public void testCheckFunctions()
{
boolean returnval=false;

returnval=test1();
if(returnval)
{
returnval=test2();
}
if(returnval)
{
returnval=test3();
}
}

You can exit the test if any of the above condition fails.

score 0 · Answer 3 · answered Apr 20 '23 at 07:09

Changing the tests order in jUnit5 can be done by using @MethodOrder

If you want to have a specific order, you should annotate your class to use OrderAnnotion

@TestMethodOrder(MethodOrderer.OrderAnnotation.class)
public class OrderInJunit {

    @Test
    @Order(1)
    public void runsFirst() {
        System.out.println("runs first");
    }
    @Test
    @Order(2)
    public void runsSecond() {
        System.out.println("runs second");
    }

}

If you want that one test fails consecutive tests in vanilla junit, you can do something like this:

@TestMethodOrder(MethodOrderer.OrderAnnotation.class)
// One test instance per class, to keep the state of "previousTestFailed" between each test
@TestInstance(TestInstance.Lifecycle.PER_CLASS)
public class OrderInJunit {

    private Boolean previousTestFailed = false;

    @Test
    @Order(1)
    public void runsFirst() {
        previousTestFailed = true;
        System.out.println("runs first");
        previousTestFailed = false;
    }

    @Test
    @Order(2)
    public void runsSecond() {
        Assumptions.assumeFalse(previousTestFailed,"No use to run this when \"runsFirst\" fails");
        previousTestFailed = true;
        System.out.println("runs second");
        fail("Test failure");
        previousTestFailed = false;
    }
    @Test
    @Order(3)
    public void runsThird() {
        Assumptions.assumeFalse(previousTestFailed,"No use to run this when \"runsSecond\" fails");
        previousTestFailed = true;
        System.out.println("runs second");
        previousTestFailed = false;
    }
}

This will always result in:

However, it's clear that this setup has some issues:

you change the order which you regulary don't want to do in jUnit
you create an instance per class instead of per test method
it does come with boilerplate since you have to flip the previousTestFailed boolean and you have to add the assumption in each test (except the first).

You probably want to look into a more E2E specific test framework or jUnit plugin(s).

Stopping Test's if JUnit Test fails

3 Answers3