How do I get the result of a Cucumber feature

Question

I'm trying to run Cucumber features in JUnit 5 Jupiter. I've lifted some code from the Cucumber-jvm source and adapted it for JUnit 5's TestFactory. It is working: I see my features running when I run all JUnit tests (this is Kotlin code, but the same applies to Java):

@CucumberOptions(
        plugin = arrayOf("pretty"),
        features = arrayOf("classpath:features")
)
class Behaviours {
    @TestFactory
    fun loadCucumberTests() : Collection<DynamicTest> {
        val options = RuntimeOptionsFactory(Behaviours::class.java).create()
        val classLoader = Behaviours::class.java.classLoader
        val resourceLoader = MultiLoader(classLoader)
        val classFinder = ResourceLoaderClassFinder(resourceLoader, classLoader)
        val runtime = Runtime(resourceLoader, classFinder, classLoader, options)
        val cucumberFeatures = options.cucumberFeatures(resourceLoader)
        return cucumberFeatures.map<CucumberFeature, DynamicTest> { feature ->
            dynamicTest(feature.gherkinFeature.name) {
                var reporter = options.reporter(classLoader)
                feature.run(options.formatter(classLoader), reporter, runtime)
            }
        }
    }
}

However, JUnit reports that every feature was successful, whether or not it actually was. When features fail, the results are correctly pretty-printed, but the generated DynamicTest passes. Neither gradle test nor Intellij notice the error: I have to inspect the text output.

I think I have to figure out, in the Executable passed as the second parameter to dynamicTest, what the result of the feature was, and raise an assertion when appropriate. How do I determine the result of feature or feature.gherkinFeature at that point?

And is there a way to get at the results for each scenario in the feature? Or better, is there a way to run a specific scenario, so that I can create a DynamicTest for each scenario, giving me better reporting granularity in JUnit?

You can use the After hook and pass the Scenario object to it. The scenario class has getStatus() which returns passed, failed, undefined, skipped, pending or the isFailed() which returns a boolean. — Grasshopper, Nov 12 '16 at 13:00
Maybe [this question](https://stackoverflow.com/questions/35550386/cucumber-jvm-hooks-when-scenario-is-passed-or-failed/35553304#35553304) could help you. — troig, Nov 14 '16 at 07:35
What type is `reporter` in your example? I haven't looked into Junit5 much, but for the junit4 integration, this should be an `JunitReporter`, which forwards the information to a junit `RunNotifier`. — Jörn Horstmann, Nov 15 '16 at 12:54
Back on this after months. Creating a Junit5 reporter has broken the back of this. I may be able to get this finished next week, time permitting. — Paul Hicks, Feb 22 '17 at 02:20
Please note that Cucumber does not yet support JUnit 5: https://github.com/cucumber/cucumber-jvm/issues/1149 — Marit, Jan 12 '18 at 08:32

score 1 · Accepted Answer · answered Feb 23 '17 at 03:44

To record the result of a Cucumber scenario as a JUnit5, I found it easiest to implement a JunitLambdaReporter, which is essentially a simpler version of the existing JunitReporter. Once you have a reporter that remembers what the current scenario is, then you can create a @TestFactory that uses this logic:

return dynamicTest(currentScenario.getName(), () -> {
  featureElement.run(formatter, reporter, runtime);
  Result result = reporter.getResult(currentScenario);

  // If the scenario is skipped, then the test is aborted (neither passes nor fails).
  Assumptions.assumeFalse(Result.SKIPPED == result);

  Throwable error = result.getError();
  if (error != null) {
    throw error;
  }
});

How do I get the result of a Cucumber feature

1 Answers1