JUnit test with expected exception does not behave as anticipated

Question

I came accross a problem with JUnit 4.12 I cannot explain when testing a method in my code which should throw a specific exception. I'm using the @Test annotation with (expected = MyException.class). MyException is simply derived from Exception.

Due to an programming error (missing 'throw' clause) this exception is not thrown. Because of the test case setup including mocked classes using jmockit 1.29, later in the method a NullPointerException is thrown.

Example classes shwoing the basic class layout:

package l00tr.test;

public class MyException extends Exception {
    private static final long serialVersionUID = 1L;
    public MyException(String msg) {
        super(msg);
    }
}

package l00tr.test;

import java.util.Map;
import java.util.Set;

public class MyClass {

    private Set<String> testSet;
    private Map<String, String> testMap;
    private boolean condition;

    private MyClass() {
        this.testSet = null;
        this.testMap = null;
        this.condition = false;
    }
    private MyClass(Set<String> testSet, Map<String, String> testMap,
            boolean condition) {
        this.testSet = testSet;
        this.testMap = testMap;
        this.condition = condition;
    }
    public void method() throws MyException {
        if (testSet.size() < 2) {
            throw new MyException("test set");
        }
        if (!condition) {
            // FORGOTTEN: throw
            new MyException("test");
        }
        System.out.print("set size: " + testSet.size());
        System.out.print("some more stuff succeeding");
        System.out.print("map size: " + testMap.size()); // NullPointerException
    }
    public static final class Builder {
        private Set<String> testSet;
        private Map<String, String> testMap;
        private boolean condition;
        private Builder() {
            this.testSet = null;
            this.testMap = null;
            this.condition = false;
        }
        public static Builder newBuilder() {
            return new Builder();
        }
        public Builder testSet(Set<String> testSet) {
            this.testSet = testSet;
            return this;
        }
        public Builder testMap(Map<String, String> testMap) {
            this.testMap = testMap;
            return this;
        }
        public Builder condition(boolean condition) {
            this.condition = condition;
            return this;
        }
        public MyClass build() {
            return new MyClass(testSet, testMap, condition);
        }
    }
}

So I would expect the test to fail. But it did succeed. Removing the "(expected = ...)" part of the Test annotation results in the test failing with the NullPointerException as to expect.

package l00tr.test;

import java.util.Set;
import org.junit.Test;
import mockit.Expectations;
import mockit.Mocked;

public class MyClassTests {
    @Mocked
    private Set<String> testSet;

    @Test(expected = MyException.class)
    public void testMethod() throws MyException {
        new Expectations() {
            {
                testSet.size();
                result = 3;
            }
        };
        MyClass mc = MyClass.Builder.newBuilder().condition(false)
                .testSet(testSet).build();
        mc.method();
    }
}

Jacoco helped to spot this, otherwise I would have wrongly assumed everything to be fine. I also checked the open issues for junit 4.12 but did not find a matching issue.

Why does the test succeed? Any ideas?

Update:

I improved the example to show the relevant details of my code. Unfortunately, the code pasted above does behave as expected, i.e. showing the error:

java.lang.Exception: Unexpected exception, expected<l00tr.test.MyException> but was<java.lang.NullPointerException>

My specific development environment setup may play a role, so here are some details:

gradle 3.2.1 with active daemon
junit 4.12
jmockit 1.29
eclipse Neon.1a Release (4.6.1)
java version "1.8.0_112" 64 bit
Windows 10

Show me your full test. Are you extending from TestCase ? http://stackoverflow.com/a/5550830/912829 — ACV, Dec 15 '16 at 16:13
@ACV Thank you, I've found that answer as well. I'm not (knowingly) extending TestCase. The code above is simplified to fit in here, but I will extend it to better show my code structure. — AntiTiming, Dec 15 '16 at 16:21
If using Junit 4 annotation, extending testcase not necessary. I cannot reproduce this problem with Junit 4, however, so in summarizing the code to post on stack overflow - something is missing. The common "false positive" problem with annotation based exception checking with Junit4 was the case where an exception you _didn't_ expect to happen satisfies your annotation, rather than the one you were really trying to test. — Jim Weaver, Dec 15 '16 at 16:21
from JUnit 4 you can use `@Rule` with `ExpectedException` to test exceptions — Jobin, Dec 16 '16 at 05:25
@JimWeaver You've been right! I've found the reason which comes down to the common "false positive" problem with annotation based exception checking with Junit4 you've mentioned. Thank you! — AntiTiming, Dec 16 '16 at 06:41

score 0 · Accepted Answer · answered Dec 16 '16 at 06:45

After the helpful comments, I've found out that the problem could be narrowed down to the "'false positive' problem with annotation based exception checking with Junit4":

After checking again the full stacktrace and code including an abstract superclass, I've found a try/catch converting any exception to MyException after logging. So from tool and code perspective, everything works as to expect.

JUnit test with expected exception does not behave as anticipated

1 Answers1