21

I would like to stop and mark as failed too long running junit tests (executed within Maven 3 build). I am aware of three ways of doing it:

1) Using Test annotation with timeout parameter:

@Test(timeout=100)
public void testWithTimeout() {
    ...
}

2) Using Rule annotation:

@Rule
public Timeout globalTimeout = new Timeout(100);

3) Configuring maven-surefire-plugin using following options:

forkedProcessTimeoutInSeconds=1
reuseForks=false

The clue is 1) and 2) requires to change each test (it hurts when you have many thousands). 3) solution is not acceptable as within many modules first test starts the context which is used by the test - tests performance would drastically decrease.

Do you have any other ideas how to achieve that? Any tricky solution (not involving patching JUnit :))?

Piotr Oktaba
  • 775
  • 1
  • 4
  • 14
  • My first question would be, what are your unit tests doing that result in so many timeouts? It sounds like you may actually be performing integration or acceptance tests. – Thorn G Sep 29 '14 at 20:19
  • Exactly, you are right. Please note, that I didn't mentioned that I am using unit tests. JUnit is used as a base framework for integration and functional tests. – Piotr Oktaba Sep 29 '14 at 20:22
  • 2
    No, but Maven generally recommends Failsafe for integration/functional tests -- you may find better options there. – Thorn G Sep 29 '14 at 20:53
  • Failsafe is almost the same as surefire and has the same options regarding forks. – Piotr Oktaba Sep 29 '14 at 20:59
  • 2
    Provide a [Java agent](http://docs.oracle.com/javase/7/docs/api/java/lang/instrument/package-summary.html) to your JVM that adds appropriate annotations to your test classes via [`ClassFileTransformer`](http://docs.oracle.com/javase/7/docs/api/java/lang/instrument/ClassFileTransformer.html). – Gerold Broser Sep 29 '14 at 21:20
  • The Rule globalTimeout doesn't require changing every test function - it works globally for all tests in a class. Did you mean it requires changing every TestClass ? – Gonen I Oct 01 '14 at 01:16
  • Have you tried forking tests with "surefire.timeout" property set? See http://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html – Andrés Oviedo Oct 27 '14 at 12:20
  • Is there a simple solution here of defining a `static final` variable for the timeout duration in some utility class, and just use that static value in all the annotations? This would allow for wide-ranging changes. Multiple variables could be defined for different cases. – George Pantazes Nov 05 '18 at 22:41

4 Answers4

4

You can try to define your own test runner by writing a class which extends BlockJunit4ClassRunner and fail if the test execution exceeds the defined timeout. Then annotate your test classes with @RunWith(CustomizedTestRunner.class) You still need to modify the classes but the timeout value can be specified in a single location.

user1048931
  • 149
  • 1
  • 4
0

Actually The Timeout Rule applies the same timeout to all test methods in a class.

If you don't want to add it to every Test class, you can define it once in a Test base class.

For something that doesn't require a change to all classes, you could implement a Junit custom suite runner ( See second code example )

Just for fun how about this hacky solution using the deprecated Thread.stop method?

Before function starts a watcher thread for every test, which after a timeout kills the Junit test thread.

If the test completes, cleanup kills the watcher thread.

Works for this little demo, not sure if this would work on production scale.

import static org.junit.Assert.*;

import org.junit.After;
import org.junit.Before;
import org.junit.Test;

// You could also make this a base class for test classes
public class TimeoutTest {

    Thread watcherThread ;
    Thread junitTestThread;
    final static int testTimeout = 2000;

    @Before
    public void myInit()
    {
        junitTestThread = Thread.currentThread();
        watcherThread = new Thread()
        {
            @Override 
            public void run()
            {
                    try {
                        Thread.sleep(testTimeout);
                        junitTestThread.stop();
                    } catch (InterruptedException e) {
                        // TODO Auto-generated catch block
                        e.printStackTrace();
                    }
            }
        };
        watcherThread.start();
    }

    @After
    public void myCleanup() throws InterruptedException
    {
        watcherThread.stop();
    }

    @Test
    public void testPassingFastTest() throws InterruptedException {
        Thread.sleep(1000);
    }

    @Test
    public void testFailingSlowTest() throws InterruptedException {
        Thread.sleep(3000);
    }

}

Or to do this for several test classes using a suite:

import java.util.Arrays;

import org.junit.runner.Description;
import org.junit.runner.RunWith;
import org.junit.runner.notification.RunListener;
import org.junit.runner.notification.RunNotifier;
import org.junit.runners.Suite;
import org.junit.runners.Suite.SuiteClasses;
import org.junit.runners.model.InitializationError;
import org.junit.runners.model.RunnerBuilder;

@RunWith(AllTests.class)
public class AllTests extends Suite{

    Thread watcherThread ;
    Thread junitTestThread;
    final static int testTimeout = 70;

    public AllTests(final Class<?> clazz) throws InitializationError {
        super(clazz, findClasses());
      }

    private static Class<?>[] findClasses() {
        // You could write code here to get the list of all test classes from specific directories
        return new  Class<?>[] {TimeoutTest.class,TimeoutTest2.class};
      }


    @Override
    public void run(final RunNotifier notifier) {


      notifier.addListener(new RunListener() {
        @Override
        public void testStarted(final Description description) {            
            System.out.println("Before test " + description.getDisplayName());
            junitTestThread = Thread.currentThread();
            watcherThread = new Thread()
            {
                @Override 
                public void run()
                {
                        try {
                            Thread.sleep(testTimeout);
                            junitTestThread.stop();
                        } catch (InterruptedException e) {
                            // TODO Auto-generated catch block
                            e.printStackTrace();
                        }
                }
            };
            watcherThread.start();

        }

        @Override
        public void testFinished(final Description description) {
            System.out.println("After test " + description.getDisplayName());
            watcherThread.stop();

        }

      }
    );

    super.run(notifier);


    }      
}
Gonen I
  • 5,576
  • 1
  • 29
  • 60
  • Also, if your test classes have a common base class, you can add this code to the Before and After of that class – Gonen I Oct 01 '14 at 01:02
0

Maybe not quite the solution you are looking for, but if you have your tests running in a continuous build server, there is usually a global timeout for the job. e.g. for Jenkins

Cfx
  • 2,272
  • 2
  • 15
  • 21
0

@Test and @Rule annotations only work with JUnit4. For JUnit3, you have to manage a new Thread manually to set a timeout:

public void testWithTimeout() throws InterruptedException, TimeoutException {
    final int factorialOf = 1 + (int) (30000 * Math.random());
    System.out.println("computing " + factorialOf + '!');

    Thread testThread = new Thread() {
        public void run() {
            System.out.println(factorialOf + "! = " + Utils.computeFactorial(factorialOf));
        }
    };

    testThread.start();
    Thread.sleep(1000);
    testThread.interrupt();

    if (testThread.isInterrupted()) {
        throw new TimeoutException("the test took too long to complete");
    }
}

Source and more info: https://netbeans.org/kb/docs/java/junit-intro.html#Exercise_24

Or a more compact solution is to use a Callable with a Future and ExecutorService. Info here: How can I wrap a method so that I can kill its execution if it exceeds a specified timeout?

Mr-IDE
  • 7,051
  • 1
  • 53
  • 59