Is there a way to make integration tests fail quickly when middleware fails?

Question

Our test environment has a variety of integration tests that rely on middleware (CMS platform, underlying DB, Elasticsearch index).

They're automated and we manage our middleware with Docker, so we don't have issues with unreliable networks. However, sometimes our DB crashes and our test fails.

The problem is that the detection of this failure is through a litany of org.hibernate.exception.JDBCConnectionException messages. These come about via a timeout. When that happens, we end up with hundreds of tests failing with this exception, each one taking many seconds to fail. As a result, it takes an age for our tests to complete. Indeed, we generally just kill these builds manually when we realise they are done.

My question: In a Maven-driven Java testing environment, is there a way to direct the build system to watch out for specific kinds of Exceptions and kill the whole process, should they arrive (or reach some kind of threshold)?

We could watchdog our containers and kill the build process that way, but I'm hoping there's a cleaner way to do it with maven.

Answer 1

If you use TestNG instead of JUnit, there are other possibilities to define tests as dependent on other tests.

For example, like others mentioned above, you can have a method to check your database connection and declare all other tests as dependent on this method.

@Test
public void serverIsReachable() {}

@Test(dependsOnMethods = { "serverIsReachable" })
public void queryTestOne() {}

With this, if the serverIsReachable test fails, all other tests which depends on this one will be skipped and not marked as failed. Skipped methods will be reported as such in the final report, which is important since skipped methods are not necessarily failures. But since your initial test serverIsReachable failed, the build should fail completely. The positive effect is, that non of your other tests will be executed, which should fail very fast.

You could also extend this logic with groups. Let's say you're database queries are used by some domain logic tests afterwards, you can declare each database test with a group, like

@Test(groups = { "jdbc" })
public void queryTestOne() {}

and declare you domain logic tests as dependent on these tests, with

@Test(dependsOnGroups = { "jdbc.* })
public void domainTestOne() {}

TestNG will therefore guarantee the order of execution for your tests.

Hope this helps to make your tests a bit more structured. For more infos, have a look at the TestNG dependency documentation.

Answer 2

I realize this is not exactly what you are asking for, but could help none the less to speed up the build:

JUnit assumptions allow to let a test pass when an assumption fails. You could have an assumption like assumeThat(db.isReachable()) that would skip those tests when a timeout is reached.

In order to actually speed things up and to not repeat this over and over, you could put this in a @ClassRule:

A failing assumption in a @Before or @BeforeClass method will have the same effect as a failing assumption in each @Test method of the class.

Of cause you would then have to mark your build as unstable via another way, but that should be easily doable.

Answer 3

I don't know if you can fail-fast the build itself, or even want to - since the administrative aspects of the build may not then complete, but you could do this:

In all your test classes that depend on the database - or the parent classes, because something like this is inheritable - add this:

@BeforeClass
public void testJdbc() throws Exception {
    Executors.newSingleThreadExecutor()
    .submit(new Callable() {
        public Object call() throws Exception {
            // execute the simplest SQL you can, eg. "SELECT 1"
            return null;
        }
    })
    .get(100, TimeUnit.MILLISECONDS);
}

If the JDBC simple query fails to return within 100ms, the entire test class won't run and will show as a "fail" to the build.

Make the wait time as small as you can and still be reliable.

Answer 4

One thing you could do is to write a new Test Runner which will stop if such an error occurs. Here is an example of what that might look like:

import org.junit.internal.AssumptionViolatedException;
import org.junit.runner.Description;
import org.junit.runner.notification.RunNotifier;
import org.junit.runners.BlockJUnit4ClassRunner;
import org.junit.runners.model.FrameworkMethod;
import org.junit.runners.model.InitializationError;
import org.junit.runners.model.Statement;

public class StopAfterSpecialExceptionRunner extends BlockJUnit4ClassRunner {

    private boolean failedWithSpecialException = false;

    public StopAfterSpecialExceptionRunner(Class<?> klass) throws InitializationError {
        super(klass);
    }

    @Override
    protected void runChild(final FrameworkMethod method, RunNotifier notifier) {
        Description description = describeChild(method);
        if (failedWithSpecialException || isIgnored(method)) {
            notifier.fireTestIgnored(description);
        } else {
            runLeaf(methodBlock(method), description, notifier);
        }
    }

    @Override
    protected Statement methodBlock(FrameworkMethod method) {
        return new FeedbackIfSpecialExceptionOccurs(super.methodBlock(method));
    }

    private class FeedbackIfSpecialExceptionOccurs extends Statement {

        private final Statement next;

        public FeedbackIfSpecialExceptionOccurs(Statement next) {
            super();
            this.next = next;
        }

        @Override
        public void evaluate() throws Throwable {
            boolean complete = false;
            try {
                next.evaluate();
                complete = true;
            } catch (AssumptionViolatedException e) {
                throw e;
            } catch (SpecialException e) {
                StopAfterSpecialExceptionRunner.this.failedWithSpecialException = true;
                throw e;
            }
        }
    }
}

Then annotate your test classes with @RunWith(StopAfterSpecialExceptionRunner.class).

Basically what this does is that it checks for a certain Exception (here it's SpecialException, an Exception I wrote myself) and if this occurs it will fail the test that threw that and skip all following Tests. You could of course limit that to tests annotated with a specific annotation if you liked.

It is also possible, that a similar behavior could be achieved with a Rule and if so that may be a lot cleaner.