How to verify that an exception was not thrown

Question

In my unit test using Mockito I want to verify that NullPointerException was not thrown.

public void testNPENotThrown{
    Calling calling= Mock(Calling.class);
    testClass.setInner(calling);
    testClass.setThrow(true);

    testClass.testMethod();

    verify(calling, never()).method();
}

My test set up the testClass, setting the Calling object and the property so that the method will throw a NullPointerException.

I verify that the Calling.method() is never called.

public void testMethod(){
    if(throw) {
        throw new NullPointerException();
    }

    calling.method();
}

I want to have a failing test because it throws a NullPointerException, and then I want to write some code to fix this.

What I have noticed is that the test always passes as the exception is never thrown up the the test method.

score 20 · Accepted Answer · edited Mar 29 '20 at 13:08

tl;dr

post-JDK8 : Use AssertJ or custom lambdas to assert exceptional behaviour.
pre-JDK8 : I will recommend the old good try-catch block. (Don't forget to add a fail() assertion before the catch block)

Regardless of Junit 4 or JUnit 5.

the long story

It is possible to write yourself a do it yourself try-catch block or use the JUnit tools (@Test(expected = ...) or the @Rule ExpectedException JUnit rule feature).

But these ways are not so elegant and don't mix well readability wise with other tools. Moreover, JUnit tooling does have some pitfalls.

The try-catch block you have to write the block around the tested behavior and write the assertion in the catch block, that may be fine but many find that this style interrupts the reading flow of a test. Also, you need to write an Assert.fail at the end of the try block. Otherwise, the test may miss one side of the assertions; PMD, findbugs or Sonar will spot such issues.
The @Test(expected = ...) feature is interesting as you can write less code and then writing this test is supposedly less prone to coding errors. But this approach is lacking in some areas.
- If the test needs to check additional things on the exception like the cause or the message (good exception messages are really important, having a precise exception type may not be enough).
- Also as the expectation is placed around in the method, depending on how the tested code is written then the wrong part of the test code can throw the exception, leading to false-positive test and I'm not sure that PMD, findbugs or Sonar will give hints on such code.
```
@Test(expected = WantedException.class)
public void call2_should_throw_a_WantedException__not_call1() {
    // init tested
    tested.call1(); // may throw a WantedException

    // call to be actually tested
    tested.call2(); // the call that is supposed to raise an exception
}
```
The ExpectedException rule is also an attempt to fix the previous caveats, but it feels a bit awkward to use as it uses an expectation style, EasyMock users know very well this style. It might be convenient for some, but if you follow Behaviour Driven Development (BDD) or Arrange Act Assert (AAA) principles the ExpectedException rule won't fit in those writing style. Aside from that it may suffer from the same issue as the @Test way, depending on where you place the expectation.
```
@Rule ExpectedException thrown = ExpectedException.none()

@Test
public void call2_should_throw_a_WantedException__not_call1() {
    // expectations
    thrown.expect(WantedException.class);
    thrown.expectMessage("boom");

    // init tested
    tested.call1(); // may throw a WantedException

    // call to be actually tested
    tested.call2(); // the call that is supposed to raise an exception
}
```
Even the expected exception is placed before the test statement, it breaks your reading flow if the tests follow BDD or AAA.

Also, see this comment issue on JUnit of the author of ExpectedException. JUnit 4.13-beta-2 even deprecates this mechanism:

Pull request #1519: Deprecate ExpectedException

The method Assert.assertThrows provides a nicer way for verifying exceptions. In addition, the use of ExpectedException is error-prone when used with other rules like TestWatcher because the order of rules is important in that case.

So these above options have all their load of caveats, and clearly not immune to coder errors.

There's a project I became aware of after creating this answer that looks promising, it's catch-exception.

As the description of the project says, it let a coder write in a fluent line of code catching the exception and offer this exception for the latter assertion. And you can use any assertion library like Hamcrest or AssertJ.

A rapid example taken from the home page :
```
// given: an empty list
List myList = new ArrayList();

// when: we try to get the first element of the list
when(myList).get(1);

// then: we expect an IndexOutOfBoundsException
then(caughtException())
        .isInstanceOf(IndexOutOfBoundsException.class)
        .hasMessage("Index: 1, Size: 0") 
        .hasNoCause();
```
As you can see the code is really straightforward, you catch the exception on a specific line, the then API is an alias that will use AssertJ APIs (similar to using assertThat(ex).hasNoCause()...). At some point the project relied on FEST-Assert the ancestor of AssertJ. EDIT: It seems the project is brewing a Java 8 Lambdas support.

Currently, this library has two shortcomings :
- At the time of this writing, it is noteworthy to say this library is based on Mockito 1.x as it creates a mock of the tested object behind the scene. As Mockito is still not updated this library cannot work with final classes or final methods. And even if it was based on Mockito 2 in the current version, this would require to declare a global mock maker (inline-mock-maker), something that may not what you want, as this mock maker has different drawbacks that the regular mock maker.
- It requires yet another test dependency.
These issues won't apply once the library supports lambdas. However, the functionality will be duplicated by the AssertJ toolset.

Taking all into account if you don't want to use the catch-exception tool, I will recommend the old good way of the try-catch block, at least up to the JDK7. And for JDK 8 users you might prefer to use AssertJ as it offers may more than just asserting exceptions.

With the JDK8, lambdas enter the test scene, and they have proved to be an interesting way to assert exceptional behaviour. AssertJ has been updated to provide a nice fluent API to assert exceptional behaviour.

And a sample test with AssertJ :

@Test
public void test_exception_approach_1() {
    ...
    assertThatExceptionOfType(IOException.class)
            .isThrownBy(() -> someBadIOOperation())
            .withMessage("boom!"); 
}

@Test
public void test_exception_approach_2() {
    ...
    assertThatThrownBy(() -> someBadIOOperation())
            .isInstanceOf(Exception.class)
            .hasMessageContaining("boom");
}

@Test
public void test_exception_approach_3() {
    ...
    // when
    Throwable thrown = catchThrowable(() -> someBadIOOperation());

    // then
    assertThat(thrown).isInstanceOf(Exception.class)
                      .hasMessageContaining("boom");
}

With a near-complete rewrite of JUnit 5, assertions have been improved a bit, they may prove interesting as an out of the box way to assert properly exception. But really the assertion API is still a bit poor, there's nothing outside assertThrows.
```
@Test
@DisplayName("throws EmptyStackException when peeked")
void throwsExceptionWhenPeeked() {
    Throwable t = assertThrows(EmptyStackException.class, () -> stack.peek());

    Assertions.assertEquals("...", t.getMessage());
}
```
As you noticed assertEquals is still returning void, and as such doesn't allow chaining assertions like AssertJ.

Also if you remember name clash with Matcher or Assert, be prepared to meet the same clash with Assertions.

I'd like to conclude that today (2017-03-03) AssertJ's ease of use, discoverable API, the rapid pace of development and as a de facto test dependency is the best solution with JDK8 regardless of the test framework (JUnit or not), prior JDKs should instead rely on try-catch blocks even if they feel clunky.

score 6 · Answer 2 · edited Nov 28 '22 at 16:13

If I understood correctly, you want something like this:

@Test(expected = NullPointerException.class)
public void testNPENotThrown {
    Calling calling= Mock(Calling .class);
    testClass.setInner(calling);
    testClass.setThrow(true);

    testClass.testMethod();

    verify(calling, never()).method();
    Assert.fail("No NPE");
}

but by the naming of the test "NPENotThrown" I would expect a test like this:

public void testNPENotThrown {
    Calling calling= Mock(Calling .class);
    testClass.setInner(calling);
    testClass.setThrow(true);

    testClass.testMethod();
    try {
        verify(calling, never()).method();
        Assert.assertTrue(Boolean.TRUE);
    } catch(NullPointerException ex) {
        Assert.fail(ex.getMessage());
    }
}

I don't think you need the Assert.fail("No NPE"); because you already have the annotation in the first block. The annotation will fail the test if a NullPointer is not thrown. — n3wb, Dec 03 '14 at 16:14

Mark Chorley · Answer 3 · 2013-07-02T23:16:17.367

Another approach might be to use try/catch instead. It's a bit untidy but from what I understand this test is going to be short-lived anyway as it's for TDD:

@Test
public void testNPENotThrown{
  Calling calling= Mock(Calling.class);
  testClass.setInner(calling);
  testClass.setThrow(true);

  try{
    testClass.testMethod();
    fail("NPE not thrown");
  }catch (NullPointerException e){
    //expected behaviour
  }
}

EDIT: I was in a hurry when I wrote this. What I mean by 'this test is going to be short-lived anyway as it's for TDD' is that you say you are going to write some code to fix this test straight away, so it will never throw a NullPointerException in the future. Then you might as well delete the test. Consequently it's probably not worth spending a lot of time writing a beautiful test (hence my suggestion :-))

More generally:

Starting off with a test to assert that (for example) the return value of a method is not null is an established TDD principle, and checking for a NullPointerException (NPE) is one possible way of going about this. However your production code is presumably not going to have a flow where a NPE is thrown. You are going to check for null and then do something sensible, I imagine. That would make this particular test redundant at that point as it will be checking an NPE is not thrown, when in fact it can never happen. You might then replace it with a test which verifies what does happen when a null is encountered: returns a NullObject for example, or throws some other type of exception, whatever is appropriate.

There is of course no requirement that you delete the redundant test, but if you don't, it will sit there making each build slightly slower, and causing each developer who reads the test to wonder; "Hmm, an NPE? Surely this code can't throw an NPE?". I have seen a lot of TDD code where the test classes have a lot of redundant tests like this. If time permits it pays to review your tests every so often.

score 0 · Answer 4 · answered Jul 02 '13 at 13:06

0

Generally each test case runs with a new instance so setting an instance variable will not help. So make 'throw' variable static if not.

answered Jul 02 '13 at 13:06

Juned Ahsan

67,789
12
98
136

How to verify that an exception was not thrown

4 Answers4

Linked