6

We are trying to use Jenkins as our CI/CD solution, and want ability to fail a deployment if our TestNG tests do not meet a set pass rate threshold.

Currently our Jenkins job executes our TestNG tests via maven, thus any test failure marks the build as failed. I.e. anything less than 100% pass rate, will fail the build job.

We want ability to make it configurable - i.e. fail the test job if pass rate is below, say, 98%.

Is this something i can control or configure inside the Jenkins job via some plugin, or is it something i have to code for inside TestNG listeners, i.e. onFinish() and set the return value based on the calculated pass rate?

StephenKing
  • 36,187
  • 11
  • 83
  • 112
Greg
  • 3,861
  • 3
  • 23
  • 58
  • I found that TestNG Results Plugin has "Thresholds for marking the build as Unstable/Failed:", but i can't seem to get it to work no matter what value i put it, the build always comes up as "Failed". – Greg Aug 11 '16 at 21:25
  • Possible duplicate of [How can I have Jenkins fail a build only when the number of test failures changes?](https://stackoverflow.com/questions/29103565/how-can-i-have-jenkins-fail-a-build-only-when-the-number-of-test-failures-change) – Destroyica Nov 27 '17 at 13:44

1 Answers1

1

I'm assuming there are some flakiness in your test cases that is causing you to make the build pass even when there are 2% failures? Otherwise, I'd say anything less than 100% is a build failure.

That being said, how about using Sonar?

With it, you can fail the build when the unit test success is < 98% (see some example screenshots here).

You can also fail the build on other occasions as well. For example, when the coverage drops below a certain amount or when a critical issue is found. Plus, you'll get the added bonus of a dashboard that contains all your build related metrics.

You will need to download a plugin called Build Breaker that will fail the build when a quality gate is below a certain threshold.

One thing to note is that the Sonarqube folks are recommending that you don't use this plugin. Thus, you can also do things like query the Sonar rest api and fail the build or wait till they add core support for it.

Community
  • 1
  • 1
Daniel Omoto
  • 2,431
  • 17
  • 15