What's the difference between failure and error in JUnit?

Question

I'm running JUnit tests on a large code base, and I've been realizing that sometimes I get "Errors" while other times I get "Failures". What's the difference?

Answer 1

Failures are when your test cases fail – i.e. your assertions are incorrect.

Errors are when unexpected errors/exceptions occur - i.e. while trying to actually run the test and an unexpected exception is thrown like FileNotFound, etc`.

Answer 2

If your test throws an exception which does not get bubbled up through the Assertion framework in Junit, it gets reported as an error. For example, a NullPointer, or a ClassNotFound exception will report an error:

String s = null;
s.trim();

or,

try {

    // your code
} catch(Exception e) {
    // log the exception
    throw new MyException(e);
}

Having said that, the following will report a failure:

Assert.fail("Failure here");

or,

Assert.assertEquals(1, 2);

or even:

throw new AssertionException(e);

It depends on the Junit version you are using. Junit 4- will make the distinction between a failure and an error, but Junit 4 simplifies it as failures only.

Following link provides more interesting inputs:

http://www.devx.com/Java/Article/31983/1763/page/2

Answer 3

From "Pragmatic Unit Testing in Java 8 with JUnit":

Assertions (or asserts) in JUnit are static method calls that you drop into your tests. Each assertion is an opportunity to verify that some condition holds true. If an asserted condition does not hold true, the test stops right there, and JUnit reports a test failure.

(It’s also possible that when JUnit runs your test, an exception is thrown and not caught. In this case, JUnit reports a test error.)

Answer 4

Below test explains the difference between Test Error vs Test failure.

I have commented the line which throws test error and test failure.

    @Test
    public void testErrorVsTestFailure() {

        final String sampleString = null;

        assertEquals('j', sampleString.charAt(0) );
        //above line throws test error as you are trying to access charAt() method on null reference

        assertEquals(sampleString, "jacob");
        //above line throws Test failure as the actual value-a null , is not equal to expected value-string "jacob"
        }

So Junit shows test error whenever you get an exception , and test failure when your expected result value doesn't match your actual value

Answer 5

Source class : JUnitReportReporter.java

public void generateReport(List<XmlSuite> xmlSuites, List<ISuite> suites, String defaultOutputDirectory) {
//......

            for (ITestResult tr : (Set) entry.getValue()) {
                TestTag testTag = new TestTag();

                boolean isSuccess = tr.getStatus() == 1;
                if (!(isSuccess)) {
                    if (tr.getThrowable() instanceof AssertionError)
                        ++errors;
                    else {
                        ++failures;
                    }
                }
}

As you can see below line in above method

tr.getThrowable() instanceof AssertionError

errors count is increased when it is instance of AssertionError otherwise(any Throwable) is counted as failures.

Answer 6

You are right that failures come from the AssertionErrors thrown by the JUnit assertion methods, or by throwing an AssertionError, or by throwing an exception that you declared in your @Test annotation, and Errors come from other, unexpected Exceptions. But there's an important distinction between them:

A failure means your test ran correctly, and identified a defect in your code.

An error could mean a bug in your code, but one that you weren't even testing for. It could also mean that the bug is in the test itself.

In short, a failure means you need to rewrite the code that is being tested. An error means that it may be the unit test that you need to rewrite. It may mean this even if the failure was in your code, such as a NullPointerException, because you detected a flaw that you weren't even testing for, so it might be wise to test for that.

Answer 7

Test Method Throws	Test Outcome
java.lang.AssertionError	failure
org.junit.AssumptionViolatedException	skipped
an expected exception	OK
any other exception	error
no exception	OK
exception outside of a test method	error

• The assert methods just throw AssertionExceptions.
• An exception is declared expected in the @Test annotation.
• Code outside of test methods can be stuff annotated with @Before, @BeforeClass and so on.
• The counts in the IDE reports count tests (test methods), not assertions, because the first failed assertion terminates the test.
• As a rule of thumb, if the test method terminates abnormally (not with a return statement), it is unsuccessful, and vice versa.

Answer 8

Ironically, junit and other testing related frameworks (testng, hamcrest) provide assert operations which verify condition and if it fails then "under-the-hood" a java.lang.AssertionError is being thrown, which btw extends java.lang.Error.

But it no way contradicts with answers above which are fully valid of course. So in order to mark specific test flow as failure one can throw AssertionError, however I'm not sure it's really documented in corresponding manuals, because more appropriate to use dedicated fail() API. Other kinds of Throwable will be considered as errors, not failures.

Answer 9

Basically, failures refer to unfulfilled assertions while errors are due to abnormal test execution. and I think each IDE has symbolic icons with different colors for passed, failed, and with error tests.

For more information, check this.