Unit tests with different severity

Question

I'm testing a set of classes and my unit tests so far are along the lines

1. read in some data from file X
2. create new object Y
3. sanity assert some basic properties of Y
4. assert advanced properties of Y

There's about 30 of these tests, that differ in input/properties of Y that can be checked. However, at the current project state, it sometimes crashes at #2 or already fails at #3. It should never crash at #1. For the time being, I'm accepting all failures at #4.

I'd like to e.g. see a list of unit tests that fail at #3, but so far ignore all those that fail at #4. What's the standard approach/terminology to create this? I'm using JUnit for Java with Eclipse.

@DaveNewton: thanks, that's a start, but I would like to enable/diable testing property #4 on a project-basis, not for each JUnit test seperately — Frank Meulenaar, May 13 '13 at 19:06
Conditionally ignoring tests maybe? http://stackoverflow.com/questions/1689242/conditionally-ignoring-tests-in-junit-4 — Balázs Németh, May 13 '13 at 19:10
Or is it maybe junit `@Rule` that you need? ErrorCollector or something similar that suits your needs. https://github.com/junit-team/junit/wiki/Rules — Balázs Németh, May 13 '13 at 19:13
On a project basis? Tests are already project-specific. Tests you don't want to be run should either be marked pending, or not written. You *could* use old-style test suites, or use categories and a reasonable test runner, etc., or run only tests from specific directories using Ant/Maven, etc. Fundamentally, though, meh. — Dave Newton, May 13 '13 at 19:13

score 0 · Answer 1 · answered May 13 '13 at 20:39

You need reporting/filtering on your unit test results.

jUnit itself wants your tests to pass, fail, or not run - nothing in between.

However, it doesn't care much about how those results are tied to passing/failing the build, or reported. Using tools like maven (surefire execution plugin) and some custom code, you can categorize your tests to distinguish between 'hard failures', 'bad, but let's go on', etc. But that's build validation or reporting based on test results rather than testing.

(Currently, our build process relies on annotations such as @Category(WorkInProgress.class) for each test method to decide what's critical and what's not).

score 0 · Answer 2 · answered May 14 '13 at 10:57

What I could think of would be to create assert methods that check some system property as to whether to execute the assert:

 public static void assertTrue(boolean assertion, int assertionLevel){
     int pro = getSystemProperty(...);
     if (pro >= assertionLevel){
         Assert.assertTrue(assertion);
     }
 }

Unit tests with different severity

2 Answers2