17

In our build there are certain scenarios that fail for reasons which are out of our control or take too long to debug properly. Things such asynchronous javascript etc.

Anyway the point is sometimes they work sometimes they don't, so I was thinking it would be nice to add a tag to a scenario such as @rerun_on_failure or @retry which would retry the scenarion X number of times before failing the build.

I understand this is not an ideal solution, but the test is still valuable and we would like to keep it without having the false negatives

The actual test that fails clicks on a link and expects a tracking event to be sent to a server for analytics (via javascript). Sometimes the selenium web-driver loads the next page too fast and the event does not have time to be sent.

Thanks

amleszk
  • 6,192
  • 5
  • 38
  • 43
  • 2
    Can you use wait_until? http://stackoverflow.com/questions/3876412/capybara-doesnt-recognize-dynamically-added-dom-elements – michaeltwofish Mar 23 '12 at 22:32
  • 1
    we do use wait_until for other tests but in this instance its not applicable. I will update the question as to why. Thanks – amleszk Mar 23 '12 at 22:46
  • I can't help thinking that cucumber hooks could achieve this. Possibly an around hook. However, it isn't just as simple as re-executing the scenario because you need your capybara driver to be in the right state as well. I've tried to get this working myself but I've not been able to do it yet. – iainbeeston May 26 '12 at 03:46
  • [Please find solution here using maven and cucumber rerun][1]. [1]: http://stackoverflow.com/questions/11719898/how-to-rerun-the-failed-scenarios-using-cucumber/32698207#32698207 – Sugat Mankar Sep 21 '15 at 14:51
  • There should be possible something like scenario.rerun() in the @After hook. But it is not. Very sad. – Čamo Apr 13 '21 at 20:30

3 Answers3

14

More recent versions of Cucumber have a retry flag

cucumber --retry 2

Will retry tests two times if it fails

Cyril Duchon-Doris
  • 12,964
  • 9
  • 77
  • 164
  • 1
    This should the new selected answer. Even if neomindryan's answer was great at the time, this answer makes it completely obsolete. – sergut Nov 30 '17 at 16:12
  • Note: in one of my build pipelines, I am retrying failed specs only if there are less than a given % of failed specs (the build should fail faster without retries if the code is completely broken), so in this case the --retry flag directly is not what I want since in the worst case (code completely broken) it would retry x times all the specs. – Cyril Duchon-Doris Jul 02 '19 at 13:05
  • How to implement it with @CucumberOptions in runner. Is it possible? – Čamo Mar 24 '21 at 09:17
  • Is there a documentation for it? I dont know how to implement it via CI. – Čamo Apr 13 '21 at 20:36
10

I've been considering writing something like what you're describing, but I found this:

http://web.archive.org/web/20160713013212/http://blog.crowdint.com/2011/08/22/auto-retry-failed-cucumber-tests.html

If you're tired of having to re-kick builds in your CI server because of non deterministic failures, this post is for you.

In a nutshell: he makes a new rake task called cucumber:rerun that uses rerun.txt to retry failed tests. It should be pretty easy to add some looping in there to retry at most 3x (for example).

Željko Filipin
  • 56,372
  • 28
  • 94
  • 125
neomindryan
  • 126
  • 2
  • 3
0

For cucumber + java on maven i found this command: mvn clean test -Dsurefire.rerunFailingTestsCount=2
, u must have actual version of surefire plugin, my is 3.0.0-M5. And nothing else special u even need.

Ivan Artuhin
  • 1
  • Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Feb 01 '22 at 13:33