Semantic for broken unit test

Question

I am writing unit tests for our main product and have one concern: how to differentiate

tests that failed because what they tested went wrong (a bug was found and its a non regression test for it for instance)
tests that failed because another unexpected part of the test failed (because the test is wrong or an unknown bug appeared)

For the first one, we have the jUnit Assert framework for sure, but what do we have for the second one?

Example: my unit test is testing that c() will not throw MyException, but to perform c() I need to first perform a() then b() which both can throw MyException(), so I would write:

@Test
public void testC() {
  a();
  Object forC = b();
  try {
    c(forC);
  } catch (MyException e) {
    Assert.fail("....");
  }
}

But then I need to handle the MyException that can be thrown by a or b, and also handle the fact that forC should not be null. What is the best way to do this?

Catch MyException thrown by a or b and Assert.fail, but a and b are not tested by this test so for me they shouldn't be marked as test failure when they fail. Maybe they fail later because at this time we should do b();a() not a();b();.
Let testC throws MyException, so that the test will fail with "MyException", but this is misleading because MyException will not tell that the test is wrongly written. All tests will then fail each with their own Exception. In this case I would also need to throw something like NullPointerException if forC is null, which also has no semantic.
Catch and wrap MyException thrown by a and b into a exception telling that the test might be wrong, like a TestCorruptedException. But I couldnt find such exceptions in jUnit, so they would not be recognized by jUnit as such (that's ok for me). Also I would need to know this exception from all my unit tests which are of course split into multiple modules, projects and so on...So this is doable but adds dependencies.

What would be your solution to this problem? I will likely go for the second but I'm not pleased with it as explained above.

score 5 · Accepted Answer · edited May 23 '17 at 11:56

5

The corner stone of unit testing is to test the minimum amount of code necessary, otherwise it arguably falls into the integration test space where you are looking for end-to-end functionality.

If you can prove that a() can be created and tested in it's own test class, and that b() can also be created and tested in it's own test class, then it follows that your test above can ignore the testing of a() and b() and instead use known values that won't fail. This is commonly satisfied by using mock objects.

With a() and b() created as mock objects, c() can be tested in isolation. If you find that it is not possible to test c() in isolation of a() and b() then this is an indicator that your code needs to change so that there is a separation of concerns. This is commonly satisfied by injecting the dependany of a() and b() into c().

This post on when to use mock objects in unit testing may help shed some more light on this topic.

edited May 23 '17 at 11:56

Community

1
1

answered Jul 13 '12 at 16:11

Brad

15,186
11
60
74

right, my problem is indeed that c is tightly linked to a() since their real name is startTransaction() and commit() so they have common structures and locks. Your answer tells me that i'm doing an integration test so I'm doomed and can only pray for a and b to be correct. Its too much effort to have them independent but that's for me the way to go to solve my problem. – jolivier Jul 13 '12 at 17:00
This is where declarative transaction management like [Spring's annotation driven Transaction Management](http://static.springsource.org/spring/docs/3.0.x/reference/transaction.html#transaction-declarative-annotations) really shines. Your method c() can throw away the calls to startTransaction() and commit() leaving you free to test it on it's own. c() won't care if it's in a transaction or not, you will catch any exceptions in a calling method. Here's a [bite sized tutorial](http://www.javacodegeeks.com/2011/09/spring-declarative-transactions-example.html) for interest – Brad Jul 13 '12 at 20:36

score 1 · Answer 2 · answered Jul 13 '12 at 16:15

A word for the wise -- never do this in a unit test

 try {
    c(forC);
  } catch (MyException e) {
    Assert.fail("....");
  }

This has a really annoying side-effect for anyone who will ever have to work on your code: the original cause of the exception is lost. This means the fixing developer will have go in and debug the test or remove the try/catch block to see the root cause of the exception.

If the test throws an exception and shouldn't, just let it. The test will fail and the test running developer can easily see in the console / log what caused the exception in the first place. Take the advice given by @piotrek, make your tests as simple as possible. I can't count how many times I've had to gut this construct out of malformed tests.

score 0 · Answer 3 · answered Jul 13 '12 at 16:05

famous problem: who will test our tests? short answer: you can't differentiate. you can always define dependencies between tests (in some testing frameworks). when one test fails then all tests depending on it are not executed. you will never know if your test code is correct. that's why it should be as simple as possible. also it should be executed before you fix the code - this way you can avoid a situation when you think your code is fixed while in fact test passes because it's wrong.

score 0 · Answer 4 · answered Jul 13 '12 at 16:14

Cool question ! :)

Every unit test i write follow this structure :

// Given
 setup for the test

// When
what i specifically test

// Then
my assertions

in your example, i assume that what's before the try is the setup. you need to call something to prepare for your test, that's allright.

But if you do your job well, a(); and b(); are tested elsewhere. So if there is a bug in one of these methods, other tests, their tests, will break. That's how you differentiate where it breaks.

The Given and the Then should never break. You are testing the When.

If you are not sure for a() and b(), mock them.

And, last case, if you want to differentiate the exceptions, assert the message.

Adriaan Koster · Answer 5 · 2012-07-20T11:53:26.903

0

Just don't catch exceptions but throw them from your test method. JUnit will indicate the difference between a test failure and a test error:

public void test() throws MyException {
    boolean result = instance.doStuff();
    Assert.true(result);
}

When you expect an Exception, do:

public void test() {
    try {
        instance.doStuff();
        Assert.fail("Expected MyException");
    }
    catch(MyException e) {
        // expected
    }
}

edited Jul 20 '12 at 11:53

answered Jul 13 '12 at 19:02

Adriaan Koster

15,870
5
45
60

Your expected exception approach is a little outdated, use @Test(expected = SomeException.class) for expected exceptions, its less code and less error prone -- the OP is using JUnit 4. – jonathan.cone Jul 20 '12 at 15:47

Semantic for broken unit test

5 Answers5