How to ignore multiple failed junit test programmatically?

Question

I have several Junit tests that run using my build.xml. I want that those tests that fail, they should be ignored the next time, i build my project. I know @ignore notation but how to implement it dynamically?

Example: I build my project using ant 1.9.3, out of 100 tests, 4 failed. My build.xml is so configured that it will run all tests i.e haltonfailure=false. Now i want that next time when I build, all those tests that failed last time should be ignored.

That makes no sense. If a test fails, it means the program has a problem. The program needs to be fixed, not the failed test removed. If these are valid tests, they should be executed each and every time. They will be that scarlet _F_ that the developer must wear -- forcing that developer into social isolation and shame until they get off their duff, stop playing Candy Crush, and fix the damn program! On the other hand, if the test isn't a valid test, remove it from the suite of tests. — David W., Apr 04 '14 at 20:43

score 1 · Accepted Answer · edited May 23 '17 at 12:28

This is a common-enough problem ("Conditionally ignoring tests in JUnit 4") that JUnit created an Assume class, which is like Assert but causes an AssumptionFailedException that JUnit treats as an ignored test. However, for your case, you probably want to conditionally ignore a lot of tests or test cases.

You'll need some kind of persistence and settings control, first of all. It's nice to ignore the tests on the second run, but would they be re-included on the third, or are they permanently ignored until you manually reset them? How do you reset them?

Now, as for ignoring the test directly: You could create a custom Rule, which decorates a Statement and also receives a Description that contains the method name etc. (You can think of a Statement as a Runnable that often represents a single test method and its decorations.) Firing an AssumptionViolatedException in Rule will probably get you where you need to go, simply, but you'd need to figure out some other way to identify which test is being run.

You may want to create a custom Runner, instead, which gives you a FrameworkMethod that contains everything you could possibly need to identify the test. In that case, BlockJUnit4TestRunner is a good starting point designed for subclassing, especially in its runChild method:

@Override
protected void runChild(final FrameworkMethod method, RunNotifier notifier) {
    Description description = describeChild(method);
    if (isIgnored(method)) {
        notifier.fireTestIgnored(description);
    } else {
        runLeaf(methodBlock(method), description, notifier);
    }
}

... which is overridable, and delegates to runLeaf in ParentRunner. Replace that method with one that checks against your storage to conditionally ignore the test, and you're probably set. Bear in mind, though, that runLeaf is final, and that you'll need to call something else if you want to update your storage when the test fails.

Thanks Jeff. I have created a custom runner and a Listener to achieve my objective. I am using @runwith in my test class to include my runner. When I run my test via eclipse run as -> junit test, my runner and listener are included everything works fine. But when I build my project using ANT, then only the that test runs and not the associated runner or Listener just like before there creation. Any idea how to resolve this? I am new to ant — Varun, Apr 05 '14 at 16:11

How to ignore multiple failed junit test programmatically?

1 Answers1