Java - TestNG : Why does my Assertion always passes when written within try-catch block

Question

I was trying with a simple code using org.testng.Assert to assert 2 use-cases. In the first use-case I am asserting 2 unequal values which Fail correctly.

But in the second use-case when I am asserting 2 unequal values within the try-catch block, the result is always returned as Pass

My code is as follows:

package demo;
import org.testng.Assert;
import org.testng.annotations.Test;
public class Q43710035 
{

    @Test
    public void test1() 
    {
        System.out.println("Within test1");
        int a = 12;
        int b =20;
        Assert.assertEquals(a, b);

    }

    @Test
    public void test2() 
    {
        System.out.println("Within test2");
        int a = 12;
        int b =20;

        try 
        {
            Assert.assertEquals(a, b);
        }catch(Throwable t)
        {
            System.out.println("Exception Occurred");
        }
    }
}

The result I am getting is:

Within test1
Within test2
Exception Occurred
PASSED: test2
FAILED: test1
java.lang.AssertionError: expected [20] but found [12]
at org.testng.Assert.fail(Assert.java:94)

My question are:

1. In test2() though the Assertion is failing why doesn't the execution comes out of this Test ?
2. In test2() it seems try block is failing, so the execution reaches catch block and Exception Occurred gets printed. Why should try block fail here when the assertion code is getting executed?

Answer 1

If you look at the source code of assertEquals method, you'll see that it all boils down to fail method (as do many other asserts).

public static void fail(String message) {
    if(message == null) {
        throw new AssertionError();
    } else {
        throw new AssertionError(message);
    }
}

As we can see, it throws AssertionError, which you are catching in catch(Throwable t). So JUnit has no way of telling that the test has failed, thus declaring it passed.

If catching an exception is part of your test (you are expecting an exception), I suggest you have a look at JUnit documentation: Exception testing

Answer 2

If you go through the source code of junit4, https://github.com/junit-team/junit4/blob/master/src/main/java/junit/framework/Assert.java#L71

You will get why the test1 is failing

Part#1:

/**
 * Asserts that two objects are equal. If they are not
 * an AssertionFailedError is thrown with the given message.
 */

static public void assertEquals(String message, Object expected, Object actual) {
    if (expected == null && actual == null) {
        return;
    }
    if (expected != null && expected.equals(actual)) {
        return;
    }
    failNotEquals(message, expected, actual); //It calls Part#2
}

Part#2:

static public void failNotEquals(String message, Object expected, Object actual) {
    fail(format(message, expected, actual)); //It calls Part#3 format(...) method
}

Part#3:

public static String format(String message, Object expected, Object actual) {
    String formatted = "";
    if (message != null && message.length() > 0) {
        formatted = message + " ";
    }
    return formatted + "expected:<" + expected + "> but was:<" + actual + ">";
}

So you have gotten Part#3's return message as

java.lang.AssertionError: expected [20] but found [12]

For full phase understanding of Exceptions JUnit Rule, Go through the following tutorial:

Expecting Exceptions JUnit Rule

To make an assertion that an exception was thrown with JUnit, it’s fairly common to use the try/fail/catch idiom or the expected element of the @Test annotation. Despite being more concise than the former, there is an argument that using expected doesn’t support all the cases you may want to test. The example being to perform additional testing after the exception or testing against the actual exception message.

JUnit 4.7 introduces the next progression, a @Rule that offers the best of both worlds. This articles weighs up the pros and cons of each approach and takes a closer look at the syntax of each. The try/fail/catch Idiom

The typical pattern is to catch an exception or fail explicitly if it was never thrown.

@Test
public void example1() {
    try {
        find("something");
        fail();
    } catch (NotFoundException e) {
        assertThat(e.getMessage(), containsString("could not find something"));
    }
    // ... could have more assertions here
}

which would highlight a failure in the following way.

java.lang.AssertionError: expected an exception
    at org.junit.Assert.fail(Assert.java:91)
    at bad.roboot.example.ExceptionTest.example1(ExceptionTest.java:20)
    ...

The idiom has potential advantages in that it offers the opportunity to assert against the actual exception as well as performing additional work after the expectation. Aside from the noise, the major drawback however is that its very easy to forget to include the fail call. If genuinely doing test first, where we always run the test red, this wouldn’t be a problem but all too often things slip through the net. In practice, I’ve seen far too many examples with a missing fail giving false positives.

@Test (expected = Exception.class)

Using the expected element, we can rewrite the test as follows.

@Test (expected = NotFoundException.class)
public void example2() throws NotFoundException {
    find("something");
    // ... this line will never be reached when the test is passing
}

which will result in the following failure.

java.lang.AssertionError: Expected exception: bad.robot.example.NotFoundException

Resource Link: What exactly does assertEquals check for when asserting lists?

Answer 3

Your second case is full of problems, that is why it behave so strange. Look at lines marked by numbers:

    try 
    {
        Assert.assertEquals(a, b); // 1
    }catch(Throwable t) // 2
    {
        System.out.println("Exception Occurred"); // 3
    }

1 - 'assert*' methods invocations should not be wrapped by try-catch block. They are designed to check data and handle it's invalidness by throwing Errors (!).

2 - You are trying to catch everything can be thrown in try section. Business logic of applications in common cases uses Exceptions, not Errors. Exceptions show that some problem appeared from which you can recover. Errors mostly show that some serious problem like (hard disc failure and so on) appeared and you just need to handle application's data before the app will be stopped. Exception and Error are inherit from Throwable. 'assert*' methods designed that way to throw Errors specifically not to be caught by 'catch' blocks.

3 - By not doing with caught exception you are just ignoring it. That's why your method always will be finished with success.

To correct second test you can: a) remove try-catch section from the method's body b) change 't' parameter's type from Throwable to Exception c) add 'throw t' at the end of 'catch' block

Answer 4

I too had faced the similar problem, later on when understanding Java, that, when the calling method wants to know about the exception in its called method, called method should contain 'throw exception;' statement in its Catch block.

In this way,

@Test
public void test2() 
{
    System.out.println("Within test2");
    int a = 12;
    int b =20;

    try 
    {
        Assert.assertEquals(a, b);
    }catch(Exception e)
    {
        System.out.println("Exception Occurred");
        throw e;
    }
}

TestNG will come to know that, there is an exception thrown. Then, TestNG will fail that method.
Please note, e.printStackTrace(); also wont Fail your Test case.