running a subset of JUnit @Test methods

Question

We're using JUnit 4 to test: we have classes that don't are a subclass of TestCase, and they have public methods annotated with @Test. We have one file with many @Test methods. It would be nice to be able to run a subset of them via Ant from the command line, in the style of this recipe for JUnit 3:

ant runtest -Dtest=MyTest -Dtests=testFoo,testBar

http://today.java.net/pub/a/today/2003/09/12/individual-test-cases.html

I've been trying to think of ways to achieve this with Java reflection, etc. Since there doesn't seem to be any way to "hide" @Test methods or remove their annotations at runtime, the only option seems to be using the ClassLoader's defineClass method, which seems quite difficult.

P.S. The Right Thing in this situation would be to split up the file, but are there alternatives?

Thanks for your time.

Answer 1

Since JUnit 4.12 we have @Category annotations to solve just that problem.

Answer 2

guerda's solution is good. Here's what I ended up doing (it's a mix of Luke Francl's recipe, which I linked before, and some other stuff I saw on the net):

import org.junit.runner.manipulation.Filter;
import org.junit.runner.Description;

public final class AntCLFilter extends Filter {
    private static final String TEST_CASES = "tests";
    private static final String ANT_PROPERTY = "${tests}";
    private static final String DELIMITER = "\\,";
    private String[] testCaseNames;

    public AntCLFilter() {
        super();
        if (hasTestCases()) testCaseNames = getTestCaseNames();
    }

    public String describe() {
        return "Filters out all tests not explicitly named in a comma-delimited list in the system property 'tests'.";
    }

    public boolean shouldRun(Description d) {
        String displayName = d.getDisplayName();
        // cut off the method name:
        String testName = displayName.substring(0, displayName.indexOf('('));
        if (testCaseNames == null) return true;

        for (int i = 0; i < testCaseNames.length; i++)
            if (testName.equals(testCaseNames[i]))
                return true;
        return false;
    }

    /**
     * Check to see if the test cases property is set. Ignores Ant's
     * default setting for the property (or null to be on the safe side).
     **/
    public static boolean hasTestCases() {
        return
            System.getProperty( TEST_CASES ) == null ||
            System.getProperty( TEST_CASES ).equals( ANT_PROPERTY ) ?
            false : true;
    }

    /**
     * Create a List of String names of test cases specified in the
     * JVM property in comma-separated format.
     *
     * @return a List of String test case names
     *
     * @throws NullPointerException if the TEST_CASES property
     * isn't set
     **/
    private static String[] getTestCaseNames() {

        if ( System.getProperty( TEST_CASES ) == null ) {
            throw new NullPointerException( "Test case property is not set" );
        }

        String testCases = System.getProperty( TEST_CASES );
        String[] cases = testCases.split(DELIMITER);

        return cases;
    }
}

import org.junit.internal.runners.*;
import org.junit.runner.manipulation.Filter;
import org.junit.runner.manipulation.NoTestsRemainException;

public class FilteredRunner extends TestClassRunner {

    public FilteredRunner(Class<?> clazz) throws InitializationError {
        super(clazz);
        Filter f = new AntCLFilter();
        try {
            f.apply(this);
        } catch (NoTestsRemainException ex) {
            throw new RuntimeException(ex);
        }
    }
}

Then I annotated my test class with:

@RunWith(FilteredRunner.class)
public class MyTest {

and put the following in my ant buildfile:

<target name="runtest"
        description="Runs the test you specify on the command line with -Dtest="
        depends="compile, ensure-test-name">
    <junit printsummary="withOutAndErr" fork="yes">
        <sysproperty key="tests" value="${tests}" />
        <classpath refid="classpath" />
        <formatter type="plain" usefile="false" />
        <batchtest>
            <fileset dir="${src}">
                <include name="**/${test}.java" />
            </fileset>
        </batchtest>
    </junit>
</target>

the key line there being the sysproperty tag.

And now I can run

ant runtest -Dtest=MyTest -Dtests=testFoo,testBar

as desired. This works with JUnit 4.1 --- in 4.4, subclass from JUnit4ClassRunner, and in 4.5 and later, subclass from BlockJUnit4ClassRunner.

Answer 3

Create your own TestClassMethodsRunner (it's not documentated or I don't find it now).
A TestClassMethodsRunner executes all TestCases and you can set up a filtered TestClassMethodsRunner.

All you have to do is override the TestMethodRunner createMethodRunner(Object, Method, RunNotifier) method. This is a simple an hacky solution:

public class FilteredTestRunner extends TestClassMethodsRunner {

    public FilteredTestRunner(Class<?> aClass) {
        super(aClass);
    }

    @Override
    protected TestMethodRunner createMethodRunner(Object aTest, Method aMethod, RunNotifier aNotifier) {
        if (aTest.getClass().getName().contains("NOT")) {
            return new TestMethodRunner(aTest, aMethod, aNotifier, null) {
                @Override
                public void run() {
                    //do nothing with this test.
                }
            };
        } else {
            return super.createMethodRunner(aTest, aMethod, aNotifier);
        }
    }

}

With this TestRunner, you execute all Tests that don't contain the string "NOT". Others will be ignored :) Just add the @RunWith annotation with your TestRunner class to your test.

@RunWith(FilteredTestRunner.class)
public class ThisTestsWillNOTBeExecuted {
   //No test is executed.
}

@RunWith(FilteredTestRunner.class)
public class ThisTestsWillBeExecuted {
   //All tests are executed.
}

In the createMethodRunner method you can check the current test against a list of tests that must be executed or introduce new criterias.

Good luck with this!

Hints for a nicer solution are appreciated!

Answer 4

There is a simpler way for the common case where you need to run only one test method, without having to go through the hassle of creating a custom Runner or Filter:

public class MyTestClass {

  public static void main(final String[] args) throws Exception {
    final JUnitCore junit = new JUnitCore();
    final String singleTest = // Get the name of test from somewhere (environment, system property, whatever you want).
    final Request req;
    if (singleTest != null) {
      req = Request.method(MyTestClass.class, singleTest);
    } else {
      req = Request.aClass(MyTestClass.class);
    }
    final Result result = junit.run(req);
    // Check result.getFailures etc.
    if (!result.wasSuccessful()) {
      System.exit(1);
    }
  }

  // Your @Test methods here.

}