JUnit 5 exlude tagged test methods within test class

Question

@SpringBootTest
@AutoConfigureMockMvc
@ExcludeTags({"no"})
public class MyClassTest {
   @Test
   public void test1() {
   }

   @Test
   @Tag("no")
   public void test2() {
   }
   ...
}

@RunWith(JUnitPlatform.class)
@SelectClasses({MyClassTest.class})
@IncludeTags({"no"})
public class MyClassTestSuiteTest {
}

Having a Spring Boot 2.3.1 project and testing some REST controllers, in a test class some of the test methods are tagged, and shall not be run, when MyClassTest is run. The annotated methods are run in a test suite (with @IncludeTags("no"). JUnit 5.6.2.

With the test suite I'm not sure it @RunWith has to be used for a test suite, or the JUnit 5 @ExtendWith is the right one? In fact, if not necessary, I don't want to mix JUnit 4 and 5, stick to JUnit 5.

Is there a way to configure simply via annotation or similar, to not run the tagged methods when MyClassTest is run? Like @ExcludeTags for test suites, but this does not work on a class like in the example.

Perhaps two test suites can be created, one with @ExludeTags("no"), one with @IncludeTags("no"). But still, how to prevent then that MyClassTest it run at all?

I don't want to create some Run Configuration in a particular IDE. The preferred way would be to use annotations or similar. Perhaps a Maven configuration would also suffice.

Perhaps on test method level execution of the particular test method can be avoided with some criteria evaluation, if the executed test class is MyClassTest, then don't run that test method.

Interesting here is, I cannot replace @RunWith(JUnitPlatform.class) simply with @ExtendWith(JUnitPlatform.class) as there is type incompatibility. Using @ExtendWith(SpringExtension.class) doesn't give me the possibility to run the class (for example with right-click on the class name, no entry to Run/Debug). But @ExtendWith replaces @RunWith in JUnit 5, what extension to use to run the test suite?

Answer 1

Create Execution Condition ExcludeTagsCondition

import java.lang.reflect.AnnotatedElement;
import java.lang.reflect.Method;
import java.util.Arrays;
import java.util.HashSet;
import java.util.Optional;
import java.util.Set;
import java.util.stream.Collectors;
import org.junit.jupiter.api.Tag;
import org.junit.jupiter.api.extension.ConditionEvaluationResult;
import org.junit.jupiter.api.extension.ExecutionCondition;
import org.junit.jupiter.api.extension.ExtensionContext;
import org.junit.platform.commons.util.AnnotationUtils;

public class ExcludeTagsCondition implements ExecutionCondition {

    private static final ConditionEvaluationResult ENABLED_IF_EXCLUDE_TAG_IS_INVALID =
            ConditionEvaluationResult.enabled(
                    "@ExcludeTags does not have a valid tag to exclude, all tests will be run");
    private static Set<String> tagsThatMustBeIncluded = new HashSet<>();

    public static void setMustIncludeTags(final Set<String> tagsThatMustBeIncluded) {
        ExcludeTagsCondition.tagsThatMustBeIncluded = new HashSet<>(tagsThatMustBeIncluded);
    }

    @Override
    public ConditionEvaluationResult evaluateExecutionCondition(
            ExtensionContext context) {
        final AnnotatedElement element = context
                .getElement()
                .orElseThrow(IllegalStateException::new);
        final Optional<Set<String>> tagsToExclude = AnnotationUtils.findAnnotation(
                context.getRequiredTestClass(),
                ExcludeTags.class
        )
        .map(a -> 
            Arrays.asList(a.value())
                    .stream()
                    .filter(t -> !tagsThatMustBeIncluded.contains(t))
                    .collect(Collectors.toSet())
        );
        if (!tagsToExclude.isPresent() || tagsToExclude.get().stream()
                .allMatch(s -> (s == null) || s.trim().isEmpty())) {
            return ENABLED_IF_EXCLUDE_TAG_IS_INVALID;
        }
        final Optional<String> tag = AnnotationUtils.findAnnotation(element, Tag.class)
                .map(Tag::value);
        if (tagsToExclude.get().contains(tag.map(String::trim).orElse(""))) {
            return ConditionEvaluationResult
                    .disabled(String.format(
                            "test method \"%s\" has tag \"%s\" which is on the @ExcludeTags list \"[%s]\", test will be skipped",
                            (element instanceof Method) ? ((Method) element).getName()
                                    : element.getClass().getSimpleName(),
                            tag.get(),
                            tagsToExclude.get().stream().collect(Collectors.joining(","))
                    ));
        }
        return ConditionEvaluationResult.enabled(
                String.format(
                        "test method \"%s\" has tag \"%s\" which is not on the @ExcludeTags list \"[%s]\", test will be run",
                        (element instanceof Method) ? ((Method) element).getName()
                                : element.getClass().getSimpleName(),
                        tag.orElse("<no tag present>"),
                        tagsToExclude.get().stream().collect(Collectors.joining(","))
                ));
    }
}

Create annotation @ExcludeTags

import static java.lang.annotation.ElementType.ANNOTATION_TYPE;
import static java.lang.annotation.ElementType.TYPE;
import static java.lang.annotation.RetentionPolicy.RUNTIME;

import java.lang.annotation.Retention;
import java.lang.annotation.Target;
import org.junit.jupiter.api.extension.ExtendWith;

@Target({ TYPE, ANNOTATION_TYPE })
@Retention(RUNTIME)
@ExtendWith(ExcludeTagsCondition.class)
public @interface ExcludeTags {
    String[] value();
}

On your test

import org.junit.jupiter.api.Tag;
import org.junit.jupiter.api.Test;
import org.springframework.boot.test.context.SpringBootTest;

@ExcludeTags({"foo", "bar"})
@SpringBootTest
class AppTest {

    @Test
    @Tag("foo")
    void test1() {
        System.out.println("test1");
    }

    @Test
    @Tag("bar")
    void test2() {
        System.out.println("test2");
    }

    @Test
    @Tag("baz")
    void test3() {
        System.out.println("test3");
    }
}

When you run the test, you should see the following output:

test method "test1" has tag "foo" which is on the @ExcludeTags list "[bar,foo]", test will be skipped

test method "test2" has tag "bar" which is on the @ExcludeTags list "[bar,foo]", test will be skipped

test3

And your test runner should show 1 test passing and 2 skipped.

Now for your test suite:

Create an annotation @MustIncludeTags

import static java.lang.annotation.ElementType.ANNOTATION_TYPE;
import static java.lang.annotation.ElementType.TYPE;
import static java.lang.annotation.RetentionPolicy.RUNTIME;

import java.lang.annotation.Retention;
import java.lang.annotation.Target;

@Target({ TYPE, ANNOTATION_TYPE })
@Retention(RUNTIME)
public @interface MustIncludeTags {
    String[] value();
}

Now setup your test suite like so:

import java.util.ArrayList;
import java.util.Arrays;
import java.util.Optional;
import java.util.stream.Collectors;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Disabled;
import org.junit.jupiter.api.Test;
import org.junit.platform.runner.JUnitPlatform;
import org.junit.platform.suite.api.SelectClasses;
import org.junit.runner.RunWith;

@RunWith(JUnitPlatform.class)
@SelectClasses({MyTestSuite.SetupTests.class, AppTest.class})
@MustIncludeTags({"foo", "bar"})
public class MyTestSuite {

    public static class SetupTests {
    
        @BeforeAll
        public static void beforeClass() {
            ExcludeTagsCondition.setMustIncludeTags(
                    Optional.ofNullable(MyTestSuite.class.getAnnotation(MustIncludeTags.class))
                            .map(MustIncludeTags::value)
                            .map(Arrays::asList)
                            .orElse(new ArrayList<>())
                            .stream()
                            .collect(Collectors.toSet())
            );
        }
    
        @Disabled
        @Test
        void testDummy() {
            // this test needs to be present for the beforeAll to run
        }
    
    }
}

When you run your test suite with the @MustIncludeTags the @ExcludedTags are overridden.

As you can see from the following test execution: