How to skip remaining steps if previous steps failed in Junit

Question

I had some integration testing cases, they ran as Junit test cases with a special category:

@Category(IntegrationTest.class)

Because they are integration testing cases, so the cost of every steps is high. Usually I will re-use some results from previous steps to reduce this cost. To make it works, I added this into them:

@FixMethodOrder(MethodSorters.NAME_ASCENDING)

Some samples like this:

@Category(IntegrationTest.class)
@FixMethodOrder(MethodSorters.NAME_ASCENDING)
public class TestAllocationPlanApi {

    @Test
    public void testStep01_verifyOrigProgram22275() {...}

    @Test
    public void testStep02_CopyProgram() {...}
}

They work well except the failure process:

If step01 failed, we don't need to run step02, but Junit still go to step02.

It is a waste and makes the test cases more complicated because you need to carefully handle those variables which are passed into step02.

I tried

-Dsurefire.skipAfterFailureCount=1

Which is discussed in another thread, but it doesn't work, the test cases still go to next step if previous steps fails.

Another annoying thing about the test cases is that Junit always resets all instance variables before every step. This forces me to use static variable to pass previous result into next step:

private static Integer contractAId;

And I have no way to run them in multiple threads.

Does anybody have good ideas to handle those things?

Thanks!

Happy new year!

glytching · Accepted Answer · 2018-01-02T19:50:35.830

You have written these as distinct tests but there are some dependencies between these tests. So, it sounds like you have split a single logical test flow across multiple test methods.

To cater for these dependencies you adopted a naming convention for the tests and instructed JUnit to run these tests in the order implied by the naming convention. In addition, you have some shared state within your test case which is being 'passed from' step to step.

This approach sounds brittle and, probably, makes the following quite difficult:

Diagnosing failures, issues
Maintaining existing steps
Adding new steps

Instructing JUnit to - somehow - stop executing subsequent tests within a test case if a prior test failed and the use of a static variable to pass previous results into the next step are both symptoms of the decision to split a single logical test across multiple @Test methods.

JUnit has no formal concept of subsequent or prior tests within a test case. This is deliberate since @Test methods are expected to be independent of each other.

So, rather than trying to implement his behaviour: stop executing subsequent tests within a test case if a prior test failed I would suggest revisiting your tests to reduce their run time, reduce costly setup time and remove this approach of splitting a single logical test flow across multiple test methods. Instead each test should be self contained; its scope should cover (a) set up, (b) execution; (c) assertion; (d) tear down.

I can see from your question that this is an integration test so it's likely that the setup, dependency management, execution etc are not simple so perhaps this approach of splitting a single logical test flow across multiple test methods is an effort to decompose a complex test flow into more digestible units. If so, then I'd recommend breaking each of these 'steps' into private methods and orchestrating them from within a single @Test method. For example:

@Test
public void test_verifyOrigProgram22275() {
    // you'll probably want to return some sort of context object from each step
    // i.e. something which allows you to (a) test whether a step has succeeded 
    // and abort if not and (b) pass state between steps

    step01_verifyOrigProgram22275();

    step02_verifyOrigProgram22275();

    ...
}

private void step01_verifyOrigProgram22275() {...}

private void step02_CopyProgram() {...}

In the beginning, I already said, they are Integration testing rather than unit test. Originally I used Jmeter to do do those jobs. But I found Jmter has more issues: (1) Hard to re-use code between test cases, (2) hard to maintain. (3) Hard to integrate with build server (4) hard to generate reports. So I decided to use JUnit to make them simple. Actually Junit help a lot except those small issues. I have to say, they are not UNIT TEST! — Justin, Jan 02 '18 at 19:57
JUnit - deliberately - has no concept of dependencies between `@Test` methods so splitting a single logical test flow across multiple `@Test` methods is a case of working against the tool. Fortunately, there's nothing about an integration test which _requires_ it to be split across multiple `@Test` methods. It could be split across multiple private/helper methods which are orchestrated by a single `@Test` method. — glytching, Jan 02 '18 at 20:05
using single @Test method is a good idea, but it is a little hard for me because: (1) every steps, I need to save API input and output, the function depends on "@Test" (2) sometimes I need to debug a single step. So I will hard code passed variables in the beginning of every steps when I am in debug mode. In your way, I have to change step01 to public and add "@Test" on it — Justin, Jan 02 '18 at 20:12
To @glytching, Thanks a lot. I changed my way to save API input and output, Now it works as your suggestion. — Justin, Jan 02 '18 at 20:37

score 1 · Answer 2 · answered Aug 01 '20 at 02:32

For my integration tests, I add the following (JUnit5, tests are ordered).

private static boolean soFarSoGood = true;
private static String failingMethod = null;
void testSoFarSoGood() throws Exception {
    Assertions.assertTrue(soFarSoGood, "Failed at method " + failingMethod);
    failingMethod = new Throwable() 
            .getStackTrace()[1] 
            .getMethodName(); 
    soFarSoGood = false;
    logger.info("Starting {}()", failingMethod);
}
void soFarSoGood() {
    soFarSoGood = true;
    logger.info("End of {}()", failingMethod);
}

@Test
@Order(10)
void first() throws Exception {
  testSoFarSoGood();
  ... test code ...
  soFarSoGood();
}
@Test
@Order(20)
void second() throws Exception {
  testSoFarSoGood();
  ... test code ...
  soFarSoGood();
}

and so on...

I couldn't make an implementation using @BeforeEach / @AfterEach work (ok... I didn't try much) but I would welcome one

How to skip remaining steps if previous steps failed in Junit

2 Answers2