11

I can perform actions on test failure by using:

@After
public void afterTest(Scenario scenario) {
    if (scenario.isFailed()) {
        /*Do stuff*/
    }
}

However some of the actions I need to perform depend on the Exception that was thrown and in what context it was thrown. Is there a way to get the Throwable that caused the test to fail? For example in JUnit I would do this by extending TestWatcher and add a rule to my tests:

@Override
protected void failed(Throwable e, Description description) {
    /*Do stuff with e*/
}

However the cucumber-junit iplementation does not allow the use of rules, so this solution would not work with Cucumber.

I don't think I need to explain why accessing a thrown exception on test failure would be useful, however I will still provide an Example:

My test environment is not always stable, so my tests might fail unexpectedly at any moment (there's no specific place I can try to catch the exception since it could occur at any time). When this happens I need the test to reschedule for another attempt, and log the incident so that we can get some good statistical data on the environment instability (when, how frequent, how long etc.)

EJS
  • 1,011
  • 5
  • 14
  • 28

8 Answers8

11

The problem with the work around suggested by Frank Escobar:

By using reflection to reach into a frameworks internals you're depending on implementation details. This is a bad practice, when ever the framework changes its implementation your code may break as you will observe in Cucumber v5.0.0.

Hooks in Cucumber are designed to manipulate the test execution context before and after a scenario. They're not made to report on the test execution itself. Reporting is cross cutting concern and best managed by using the plugin system.

For example:

package com.example;

import io.cucumber.plugin.ConcurrentEventListener;
import io.cucumber.plugin.event.EventPublisher;
import io.cucumber.plugin.event.Result;
import io.cucumber.plugin.event.Status;
import io.cucumber.plugin.event.TestCase;
import io.cucumber.plugin.event.TestCaseFinished;

public class MyTestListener implements ConcurrentEventListener {
    @Override
    public void setEventPublisher(EventPublisher publisher) {
        publisher.registerHandlerFor(TestCaseFinished.class, this::handleTestCaseFinished);
    }

    private void handleTestCaseFinished(TestCaseFinished event) {
        TestCase testCase = event.getTestCase();
        Result result = event.getResult();
        Status status = result.getStatus();
        Throwable error = result.getError();
        String scenarioName = testCase.getName();
        String id = "" + testCase.getUri() + testCase.getLine();
        System.out.println("Testcase " + id + " - " + status.name());
    }
}

When using JUnit 4 and TestNG you can activate this plugin using:

@CucumberOptions(plugin="com.example.MyTestListener")

With JUnit 5 you add it to junit-platform.properties:

cucumber.plugin=com.example.MyTestListener 

Or if you are using the CLI

--plugin com.example.MyTestListener 
M.P. Korstanje
  • 10,426
  • 3
  • 36
  • 58
5

I've implemented this method using reflections. You can't access directly to steps errors (stack trace). I've created this static method which allows you to access to "stepResults" attribute and then you can iterate and get the error and do whatever you want.

import cucumber.runtime.ScenarioImpl; 
import gherkin.formatter.model.Result;  
import org.apache.commons.lang3.reflect.FieldUtils;  
import java.lang.reflect.Field;  
import java.util.ArrayList;

@After
public void afterScenario(Scenario scenario) {
  if (scenario.isFailed())
    logError(scenario);
}


private static void logError(Scenario scenario) {
   Field field = FieldUtils.getField(((ScenarioImpl) scenario).getClass(), "stepResults", true);
   field.setAccessible(true);
   try {
       ArrayList<Result> results = (ArrayList<Result>) field.get(scenario);
       for (Result result : results) {
           if (result.getError() != null)
               LOGGER.error("Error Scenario: {}", scenario.getId(), result.getError());
       }
   } catch (Exception e) {
       LOGGER.error("Error while logging error", e);
   }
}
Frank Escobar
  • 648
  • 5
  • 15
2

You can to this by writing your own custom implementation of Formatter & Reporter interface. The empty implementation of Formatter is the NullFormatter.java which you can extend. You will need to provide implementations for the Reporter interface.

The methods which would be of interest will be the result() of the Reporter interface and possibly the done() method of Formatter. The result() has the Result object which has the exceptions.

You can look at RerunFormatter.java for clarity.

Github Formatter source

public void result(Result result) {
      //Code to create logs or store to a database etc...
      result.getError();
      result.getErrorMessage();
}

You will need to add this class(com.myimpl.CustomFormRep) to the plugin option.

plugin={"pretty", "html:report", "json:reports.json","rerun:target/rerun.txt",com.myimpl.CustomFormRep}

More details on custom formatters.

You can use the rerun plugin to get a list of failed scenarios to run again. Not sure about scheduling a run of failed tests, code to create a batch job or schedule one on your CI tool.

Grasshopper
  • 8,908
  • 2
  • 17
  • 32
1

This is the workaround for cucumber-java version 4.8.0 using reflection.

import cucumber.api.Result;
import io.cucumber.core.api.Scenario;
import io.cucumber.core.logging.Logger;
import io.cucumber.core.logging.LoggerFactory;
import io.cucumber.java.After;
import org.apache.commons.lang3.ClassUtils;
import org.apache.commons.lang3.reflect.FieldUtils;

import java.io.IOException;
import java.lang.reflect.Field;
import java.net.URL;
import java.util.ArrayList;

@After
public void afterScenario(Scenario scenario) throws IOException {
    if(!scenario.getStatus().isOk(true)){
        logError(scenario);
    }
}

private static void logError(Scenario scenario) {
    try {
        Class clasz = ClassUtils.getClass("cucumber.runtime.java.JavaHookDefinition$ScenarioAdaptor");
        Field fieldScenario = FieldUtils.getField(clasz, "scenario", true);
        fieldScenario.setAccessible(true);
        Object objectScenario =  fieldScenario.get(scenario);

        Field fieldStepResults = objectScenario.getClass().getDeclaredField("stepResults");
        fieldStepResults.setAccessible(true);

        ArrayList<Result> results = (ArrayList<Result>) fieldStepResults.get(objectScenario);
        for (Result result : results) {
            if (result.getError() != null) {
                LOGGER.error(String.format("Error Scenario: %s", scenario.getId()), result.getError());
            }
        }
    } catch (Exception e) {
        LOGGER.error("Error while logging error", e);
    }
}
Frank Escobar
  • 648
  • 5
  • 15
  • Above function is not working with cucumber-java version 5.2.0. It gives the exception of cucumber.runtime.java.JavaHookDefinition$ScenarioAdaptor class not found. – Bilal Feb 11 '20 at 07:07
  • @Bilal In the description clearly I'm specifying is for cucumber-java version 4.8.0. When the library has a major change like from 4 to 5, it's because there is no compatibility. Maybe version 5 could implement this out of the box. – Frank Escobar Feb 12 '20 at 10:13
  • Can you please guide for ver 5.2.0 as I am totally new and can't find any help regarding this. – Bilal Feb 12 '20 at 11:51
0

If you just want to massage the result being sent to the report then you can extend the CucumberJSONFormatter and override the result method like this:

public class CustomReporter extends CucumberJSONFormatter {

    CustomReporter(Appendable out) {
        super(out);
    }

    /**
     * Truncate the error in the output to the testresult.json file.
     * @param result the error result
    */
    @Override
    void result(Result result) {
        String errorMessage = null;
        if (result.error) {
            errorMessage = "Error: " + truncateError(result.error);
        }
        Result myResult = new Result(result.status, result.duration, errorMessage);
        // Log the truncated error to the JSON report
        super.result(myResult);
    }
}

Then set the plugin option to:

plugin = ["com.myimpl.CustomReporter:build/reports/json/testresult.json"]
Joman68
  • 2,248
  • 3
  • 34
  • 36
0

For cucumber-js https://www.npmjs.com/package/cucumber/v/6.0.3

import { After } from 'cucumber'

After(async function(scenario: any) {
    const exception = scenario.result.exception
    if (exception) {
        this.logger.log({ level: 'error', message: '-----------StackTrace-----------' })
        this.logger.log({ level: 'error', message: exception.stack })
        this.logger.log({ level: 'error', message: '-----------End-StackTrace-----------' })
    }
})

enter image description here

Frank Escobar
  • 648
  • 5
  • 15
  • This doesn't seem to work because exception does not exist in the scenario object with the latest version of cucumber. Any other ideas? Perhaps with reflection? – Phonesis Oct 23 '19 at 12:20
  • @Phonesis To said that you have to be sure. That `exception` attribute is `Error` type. – Frank Escobar Oct 24 '19 at 13:21
  • I am using latest version of cucumber via npm and don't get this. I just see duration and status. – Phonesis Oct 24 '19 at 13:36
  • Just an update, turns out it's the "Scenario: any" as param type that is key here. It works with any set as type. If I used HookScenarioResult as type it can't find exception. – Phonesis Oct 24 '19 at 14:43
  • @Phonesis that's right. With `HookScenarioResult` you are parsing the scenario to that object and that object doesn't have the exception. – Frank Escobar Oct 29 '19 at 10:40
0

After a lot of experimentation I now removed the Before/After-Annotations and rely on Cucumber-Events instead. They contain the TestCase (which is what the Scenario-class wraps) and a Result where you can call getError(); to get the Throwable.

Here is a simple example to get it working

import io.cucumber.plugin.EventListener;
import io.cucumber.plugin.event.EventPublisher;
import io.cucumber.plugin.event.Result;
import io.cucumber.plugin.event.Status;
import io.cucumber.plugin.event.TestCase;
import io.cucumber.plugin.event.TestCaseFinished;
import io.cucumber.plugin.event.TestCaseStarted;
import org.openqa.selenium.WebDriver;

public class TestCaseListener implements EventListener {
    @Override
    public void setEventPublisher(final EventPublisher publisher) {
        publisher.registerHandlerFor(TestCaseStarted.class, this::onTestCaseStarted);
        publisher.registerHandlerFor(TestCaseFinished.class, this::onTestCaseFinished);
    }

    public void onTestCaseStarted(TestCaseStarted event) {
        TestCase testCase = event.getTestCase();
        System.out.println("Starting " + testCase.getName());

        // Other stuff you did in your @Before-Method.
        // ...
    }

    private void onTestCaseFinished(final TestCaseFinished event) {
        TestCase testCase = event.getTestCase();
        System.out.println("Finished " + testCase.getName());

        Result result = event.getResult();
        if (result.getStatus() == Status.FAILED) {
            final Throwable error = result.getError();
            error.printStackTrace();
        }

        // Other stuff you did in your @After-Method.
        // ...
    }
}

All that's left to do is to register this class as a Cucumber-Plugin. I did this by modifying my @CucumberOptions-annotation:

@CucumberOptions(plugin = {"com.example.TestCaseListener"})

I find this much cleaner than all of this reflection-madness, however it requires a lot more code-changes.

Edit

I don't know why, but this caused a lot of tests to randomly fail in a multithreaded environment. I tried to figure it out, but now also use the ugly reflections mentioned in this thread:

public class SeleniumUtils {
private static final Logger log = LoggerFactory.getLogger(SeleniumUtils.class);

private static final Field field = FieldUtils.getField(Scenario.class, "delegate", true);
private static Method getError;

public static Throwable getError(Scenario scenario) {
    try {
        final TestCaseState testCase = (TestCaseState) field.get(scenario);
        if (getError == null) {
            getError = MethodUtils.getMatchingMethod(testCase.getClass(), "getError");
            getError.setAccessible(true);
        }
        return (Throwable) getError.invoke(testCase);
    } catch (Exception e) {
        log.warn("error receiving exception", e);
    }
    return null;
}
}
Leo Hilbert
  • 395
  • 2
  • 12
  • Rather then using `EventListener` you should use `ConcurrentEventListener`. Otherwise Cucumber will only provide you the events after the concurrent execution is over. – M.P. Korstanje Mar 14 '20 at 11:34
0

It took a bit of research and trial, but I found the solution. No need to add any plugin or listeners. Just add this in your hooks class and you are good to go.

@AfterStep
public void getErrors( Scenario scenario ) throws NoSuchFieldException, IllegalAccessException
{
    Field delegate = scenario.getClass().getDeclaredField("delegate");
    delegate.setAccessible(true);

    TestCaseState tcs = ( TestCaseState ) delegate.get(scenario);

    Field stepResults = tcs.getClass().getDeclaredField("stepResults");
    stepResults.setAccessible(true);

    ArrayList<Result> results = ( ArrayList<Result> ) stepResults.get(tcs);
    for ( Result result : results )
    {
        if ( result.getError() != null )
        {
            System.out.println(result.getError());
            //Pass it to extent report or do anything with it
        }
    }
}

Please, try this first. It'll work for sure and then upvote so that more people can use the code instead of older versions.

AKP
  • 1
  • 1