How to rerun failed features in karate?

Question

Can anyone help me to rerun failed features in karate. below are the cucumber options and runner test which is using for parallel -

@CucumberOptions(features = "classpath:features/xxxxx/crud_api",
        format = {"pretty", "html:target/cucumber","json:target/cucumber/report.json", "rerun:target/rerun/rerun.txt" })



@Test
    public void  test() throws IOException {
        Results results = KarateRunnerTest.parallel(getClass(), threadCount, karateOutputPath);
        assertTrue("there are scenario failures", results.getFailCount() == 0);
    }

Answer 1

Here is my reusable implementation using karate-1.0#retry-framework-experimental,

Results retryFailedTests(Results results) {
        System.out.println("======== Retrying failed tests ========");

        Results initialResults = results;
        List<ScenarioResult> retryResult = results.getScenarioResults().filter(ScenarioResult::isFailed)
                .parallel()
                .map(scenarioResult -> initialResults.getSuite().retryScenario(scenarioResult.getScenario()))
                .collect(Collectors.toList());
        for (ScenarioResult scenarioResult : retryResult) {
            results = results.getSuite().updateResults(scenarioResult);
        }
        return results;
    }

This java function takes care of retrying failed scenarios in parallel. You can check karate-timeline.html report to verify if the failed scenarios are retried in parallel.

Answer 2

Sharing my solution for when using the standalone JAR, implemented as a RuntimeHook (requires Karate 1.0+):

package retries;

import com.intuit.karate.core.Tag;
import com.intuit.karate.RuntimeHook;
import com.intuit.karate.core.ScenarioRuntime;
import com.intuit.karate.core.ScenarioResult;
import com.intuit.karate.core.FeatureRuntime;
import com.intuit.karate.core.Scenario;
import com.intuit.karate.core.StepResult;
import com.intuit.karate.core.Step;
import java.util.HashSet;
import java.util.List;

/**
 * RuntimeHook that implements @retries tag.
 *
 * Usage:
 *   - compile this into a jar with:
 *       javac --release 8 -cp path/to/karate.jar *.java
 *       jar cvf retries-hook.jar *.class
 *   - tag your tests like @retries=3 to retry e.g. 3 times (4 total attempts)
 *   - invoke karate tests with:
 *       java -cp path/to/karate.jar:retries-hook.jar:path/to/java" com.intuit.karate.Main path/to/tests/dir --hook retries.RetriesHook
 */
public class RetriesHook implements RuntimeHook {

    private static final HashSet<Scenario> RETRIES_ATTEMPTED = new HashSet<Scenario>();
    private static final HashSet<ScenarioResult> RETRIES_SUCCEEDED = new HashSet<ScenarioResult>();

    @Override
    public void afterScenario(ScenarioRuntime sr) {
        if (!sr.isFailed()) {
            return;
        }

        int configuredRetries = 0;
        for (Tag tag : sr.tags) {
            if ("retries".equals(tag.getName())) {
                configuredRetries = Integer.parseInt(tag.getValues().get(0));
                break;
            }
        }
        if (configuredRetries <= 0) {
            return;
        }

        for (Scenario s : RETRIES_ATTEMPTED) {
            if (s.isEqualTo(sr.scenario)) {
                // we've already kicked off retries for this Scenario
                return;
            }
        }
        String scenarioName = sr.scenario.toString();

        RETRIES_ATTEMPTED.add(sr.scenario);
        int retryAttempt = 1;
        while (retryAttempt <= configuredRetries) {
            System.out.println("Scenario " + scenarioName + " failed, attempting retry #" + retryAttempt);
            ScenarioResult retrySr = sr.featureRuntime.suite.retryScenario(sr.scenario);
            if (!retrySr.isFailed()) {
                System.out.println("Scenario " + scenarioName + " passed after " + retryAttempt + " retries");
                // Mark the original ScenarioResult as passed on retry, so it can get filtered out later in afterFeature.
                RETRIES_SUCCEEDED.add(sr.result);
                sr.featureRuntime.result.getScenarioResults().add(retrySr);
                return;
            }
            retryAttempt++;
        }
        System.out.println("Scenario " + scenarioName + " failed all " + configuredRetries + " retries");
    }

    @Override
    public void afterFeature(FeatureRuntime fr) {
        // afterScenario is called before the original ScenarioResult is saved,
        // so we can't use Suite.updateResults() :/
        // Instead, we add the passed ScenarioResult above and then filter out the
        // failed one here.
        if (fr.result.isFailed()) {
            List<ScenarioResult> scenarioResults = fr.result.getScenarioResults();
            scenarioResults.removeIf(sr -> RETRIES_SUCCEEDED.contains(sr));
            fr.result.sortScenarioResults();
        }
    }
}

Answer 3

This is not something that Karate supports, but in dev mode (using the IDE for example) you can always re-run the failed tests manually.

You seem to be using annotation options not supported by Karate, e.g. format. Read the docs for what is supported it is limited to features and tags.

EDIT - Karate 1.0 has experimental support for this: https://github.com/intuit/karate/wiki/1.0-upgrade-guide#retry-framework-experimental