1

I have a job A in Jenkins for my automated testing that is triggered if another job B build is successful. The job A run several tests. Some of the test are flaky so I would like to run them again few times and let them the chance to pass so my build won't be unstable/failed.

Is there any plugin I can use?

CSchulz
  • 10,882
  • 11
  • 60
  • 114
Benjo
  • 133
  • 1
  • 9
  • 1
    I think the answer that S. Spieker gave to you is the only that makes sense in the long run. If some of your tests are not reliable, then you better figure out why that is; and what to do in order to get them stable. – GhostCat Jun 20 '16 at 13:18

2 Answers2

1

I would suggest to fix your tests or rewrite them so they will only fail if something is broken. Maybe you can mock away the things that tend to fail. If you are depnending on a database connection, maybe you could use a sqlite or smething which is local.

But there is also a plugin which can retry a build: https://wiki.jenkins-ci.org/display/JENKINS/Naginator+Plugin

Simply install the plugin, and then check the Post-Build action "Retry build after failure" on your project's configuration page.

If you want to rerun tests in JUnit-context, take a look here: SO: How to Re-run failed JUnit tests immediately?

Community
  • 1
  • 1
S.Spieker
  • 7,005
  • 8
  • 44
  • 50
  • Thanks for replying. My tests pass most of the time. Just few of them doesn't. I can't really mock the tests because my server return and display different data in similar request. The plugin that you mentioned will eventually run all the tests again (not just the tests that failed) so it's not a solution for my problem. – Benjo Jun 20 '16 at 10:06
  • I added a link to another Question which might solve your problem. – S.Spieker Jun 20 '16 at 10:19
0

Don't know of any plugin to run just the flaky/failed tests again, only the whole build. It should be possible, I just have not found any (and don't have enough time on my hand to write one). Here's what we did on a large java project where the build was ant based:

The build itself was pretty simple (using xml as formatter inside the junit ant task):
ant clean compile test

The build also accepted a single class name as parameter (using batchtest include section inside the junit ant task):
ant -Dtest.class.pattern=SomeClassName test

At the end of the jenkins job, we used the "Execute shell" build step. The idea was to search for all test results that had errors or failures, figure out the name of the class, then run that particular test class again. The file containing the failure will be overwritten, and the test collector at the end of the build will not see the flaky test failure, during the post build steps. 

#!/bin/bash +x
cd ${WORKSPACE}
for i in $(seq 1 3); do 
    echo "Running failed tests $i time(s)"
    for file in `find -path '*/TEST-*.xml' | xargs grep 'errors\|failures' | grep '\(errors\|failures\)="[1-9]' | cut -d ':' -f 1`; do 
        class=`basename ${file} .xml | rev | cut -d '.' -f 1 | rev`
        ant -Dtest.class.pattern=${class} test
    done
done

After getting the build back under control, you definitely need to address the flaky tests. Don't let the green build fool you, there's still work to be done.

Rica Teglas
  • 1