0

I am trying to automate my build through Jenkins and facing some issues.

  1. I have some Unit test cases and some Sanity (Blockers) test in separate xml suites, and I want that if a Unit test fails the build should not be affected (should complete successfully) though I should got the failed case reports. But if any of the Sanity test case failed the build should be marked as failed.

  2. Also can I marked the build as failed if more than 70% Unit Test cases failed

Is this achievable thought some plugins or scripts, if so, Please let me know how?

I am using TestNG, Maven3 and Jenkins

Ankit S
  • 75
  • 2
  • 11

1 Answers1

0

I think this question is similar to your first one: How to Run TestNG Tests on Jenkins. have a look and let me know.

Regarding the second one it is probably technically possible, but it is never a good idea.

When you write a test you are defining the expected behavior of your code: a failing test means you found a bug!

If the 70% threshold is due to unstable tests (aka tests failing randomly) than my only advice is: live with build unstability or rewrite them (the second is way way better). What happens if the threshold "hides" a real bug?

Community
  • 1
  • 1
Manuel Spezzani
  • 1,041
  • 7
  • 15
  • I think my problem is different to the link you posted, My test cases are running fine and i dont have any problem in executing them. My problem is that the build should not failed on failure of one unit test as I have almost 500, I just want notification for them(as they may be specific to a particular fuctionality). But there are some of them which may prove blockers and the application should not be deployed if they fails. I am talking about the 'NIGHTLY CI BUILDS' here not the releases. So, I am asking if this is possible. – Ankit S Feb 22 '14 at 12:14
  • Ok, I got it. But my answer is still the same: don't do it. Ignoring tests only lead to issues. And you'll need to manually check the test report each and every morning instead let the CI Server doing it for you. I don't want to be rude, but in my opinion the real problem is that someone is committing broken code: you have to fix that bad habit, not hack the CI Server. The only (barely) acceptable workaround I can see is to run the deployment *before* running the unit tests (but after the sanity tests). This way if the code is broken you still have an updated environment AND a broken build. – Manuel Spezzani Feb 22 '14 at 18:32
  • Even I agreed to that what you are saying. But I want if different teams made some changes and at night when the NIGHTLY BUILD runs it should not fail due to a small defect. So that we can see other changes after the deployment. Your suggestion seems to be a good work around. Could you please be a little more elaborative on it, means how can I do it. I am a little new in Jenkins. Please bear me. – Ankit S Feb 24 '14 at 13:28
  • It depends on the structure of your build. A solution is to split test into two packages: the first one contains only sanity test, the second one contains only unit tests. A more elegant solution is to keep your current structure and decorate methods with a "group" annotation (http://testng.org/doc/documentation-main.html#annotations). Both ways allow you to categorize your tests in order to properly instruct testng. Hope it helps. – Manuel Spezzani Feb 24 '14 at 20:06