46

I'm using Cucumber for my tests. How do I rerun only the failed tests?

AlG
  • 14,697
  • 4
  • 41
  • 54
Kalyan
  • 529
  • 1
  • 6
  • 11

5 Answers5

63

Run Cucumber with rerun formatter:

cucumber -f rerun --out rerun.txt

It will output locations of all failed scenarios to this file.

Then you can rerun them by using

cucumber @rerun.txt
Andrei Botalov
  • 20,686
  • 11
  • 89
  • 123
  • it there anyway of putting it all in one cmd line? – Elad Benda2 Jul 22 '14 at 12:44
  • You can use a rake task for run both cucumber commands. Afterwards you can call the rake file as: rake features:jenkins_with_rerun – Pablo Gómez Jul 23 '14 at 14:47
  • 3
    You can use && to put it all in one cmd line (if you want to just start it and walk away) cucumber -f rerun --out rerun.txt && cucumber @rerun.txt – Joe Susnick Oct 30 '14 at 20:20
  • 6
    @JoeSusnick you can't use && because it won't rerun if the first command fails (which is when you need it!) You want || (to only rerun on failure) or ; (to always rerun). – Dave Burt Jan 05 '15 at 05:34
  • Also, the file is not read first, but written first. So if you would like to chain cucumber command calls, with each generating a `rerun.txt` file using that format, that won't simply work =/. You will have to alternate the filename or use some other strategy for rerunning tests quickly. – Pysis Jul 15 '16 at 17:47
  • In 2016, how would this work with Protractor since I run my tests with "protractor protractor.conf.js" ? Lastly, I'm using cucumberJS, the Javascript implementation of cucumber. – pelican Oct 12 '16 at 13:26
  • 1
    rerun.txt text holds the feature file name of the failed scenario. If i have two scenarios in a feature file and one of it fails, does the rerun do on feature level making the passed one to run again, and it is not on the scenario level, as i understood. Is there a way to identify just the failed scenario names instead of the entire feature? – Vignesh Paramasivam Jul 21 '17 at 12:24
  • how will this help when we are trying to run our tests via maven? – Aravinth Rajan Nov 19 '19 at 13:09
  • This is no longer the best answer, use the `retry` flag and pass an integer. – Anthony Chuinard Nov 20 '19 at 22:35
  • If one gets parse error like me one needs to change `IO.read('rerun.txt')` with `IO.read('rerun.txt').gsub(/\n/, ' ')` in `cucumber.yml` – Kote Jul 15 '20 at 12:17
14

Here is my simple and neat solution.

Step 1: Write your cucumber java file as mentioned below with rerun:target/rerun.txt. Cucumber writes the failed scenarios line numbers in rerun.txt as shown below. 

features/MyScenaios.feature:25
features/MyScenaios.feature:45

Later we can use this file in Step 2. Name this file as MyScenarioTests.java. This is your main file to run your tagged scenarios. If your scenarios has failed test cases, MyScenarioTests.java will write/mark them rerun.txt under target directory.

@RunWith(Cucumber.class)
@CucumberOptions(
    monochrome = true,
    features = "classpath:features",
    plugin = {"pretty", "html:target/cucumber-reports",
              "json:target/cucumber.json",
              "rerun:target/rerun.txt"} //Creates a text file with failed scenarios
              ,tags = "@mytag"
           )
public class MyScenarioTests   {

}

Step 2: Create another scenario file as shown below. Let's say this as FailedScenarios.java. Whenever you notice any failed scenario run this file. This file will use target/rerun.txt as an input for running the scenarios.

@RunWith(Cucumber.class)
@CucumberOptions(
    monochrome = true,
    features = "@target/rerun.txt", //Cucumber picks the failed scenarios from this file 
    format = {"pretty", "html:target/site/cucumber-pretty",
            "json:target/cucumber.json"}
  )
public class FailedScenarios {

}

Everytime if you notice any failed scenarios run the file in Step 2

vkrams
  • 7,267
  • 17
  • 79
  • 129
  • I tried exact same steps as you mention above. I could see that rerun.txt has been generated but still failed scenario are not getting executed – SachinB Mar 07 '18 at 11:37
  • You have to use `FailedScenarios` class to execute your tests. In other words, you need to right click on `FailedScenarios` and hit run. – vkrams Mar 08 '18 at 04:14
  • I could achieve it. I have posted my answer here. https://stackoverflow.com/questions/49132447/rerunning-the-failed-scenario-using-maven-cucumber-serenity/49168147#49168147 – SachinB Mar 08 '18 at 08:39
  • @vkrams This works for me when running from a IDE. But, when running from the command line with gradle, the `@target/rerun.txt` part isn't recognized and I get the `Undefined scenarios` message. Did you ever encounter something like this? – Mate Mrše Apr 19 '19 at 08:18
  • @MateMrše This solution works for maven project. For gradle, make sure rerun.txt written in target directory if there is atleast one failed scenario. Also make sure you are using right cucumber version 1.2.5 – vkrams Apr 23 '19 at 00:37
  • will this solution not get into a infinite loop if the script fails due to some other issue? how is that handled here? have you controlled the number of reruns anywhere? – Aravinth Rajan Nov 19 '19 at 13:08
  • 1
    I tried exactly the same steps as above, using Eclipse. I can see the rerun.txt file, which says: Login.feature:4 Presumably that means the fourth step in that particular feature file. Unfortunately, when I run the failed scenario class, I get: java.lang.IllegalArgumentException: Neither found on file system or on classpath: Not a file or directory: C:\Users\ttoth-fejel\Git\TestAutomation\Cucumber\Login.feature, No resource found for: classpath:Login.feature But if the classpath didn't include that file, the first test never could run. How do I debug this? Or add the proper classpath? – Tihamer Mar 31 '20 at 23:02
1
task cucumber() {
    dependsOn assemble, compileTestJava
    doLast {
        javaexec {
            main = "io.cucumber.core.cli.Main"
            classpath = configurations.cucumberRuntime + sourceSets.main.output + sourceSets.test.output
            args = ['--plugin', 'json:target/cucumber-reports/json/cucumber.json',
                    '--plugin', "rerun:target/rerun.txt",
                    '--glue', 'steps',
                    'src/test/resources']
        }
    }
}

task cucumberRerunFailed() {
    doLast {
        javaexec {
            main = "io.cucumber.core.cli.Main"
            classpath = configurations.cucumberRuntime + sourceSets.main.output + sourceSets.test.output
            args = ['--plugin', 'json:target/cucumber-reports/json/cucumber.json',
                    '@target/rerun.txt']
        }
    }
}
Shinigamiyuu
  • 45
  • 1
  • 7
1

I know this is old, but I found my way here first and then later found a much more up to date answer (not the accepted, Cyril Duchon-Doris' answer): https://stackoverflow.com/a/41174252/215789

Since cucumber 3.0 you can use --retry to specify how many times to retry a scenario that failed. https://cucumber.io/blog/open-source/announcing-cucumber-ruby-3-0-0/

Just tack it on to your cucumber command: cucumber ... --retry 2

Randy Eppinger
  • 1,054
  • 13
  • 23
0

You need at least version 1.2.0 in order to use the @target/rerun.txt new feature. After that just create a runner that runs at the end and uses this file. Also, if you are using Jenkins, you can put a tag on the random failures features so the build doesn't fails unless after being ran twice.

Popescu M
  • 37
  • 1
  • 9