How do I label flaky tests in junit xml syntax that jenkins uses to give us a report? jenkins gives me a nice report of tests that succeeeded and failed. I would like to know which tests are known to be flaky.
Asked
Active
Viewed 3,357 times
5
-
What makes you think JUnit can detect a "flaky" test? – kryger Sep 16 '16 at 20:54
-
Does JUnit have to do it? The question appears to be asking for the XML syntax. Maybe they plan to do it themselves. – Hakanai Nov 28 '16 at 22:27
2 Answers
2
Certainly a flaky test is something that needs to be avoided and fixed. Tests should always pass and if not they need to be worked on.
However we don't live in a perfect world and thus it might be helpful to identify tests that have failed every now and then in the past.
Since Jenkins keeps track of test results you can step through the history of a particular test case (using the "Previous/Next build" links).
Additionally there are two Jenkins plugins that might be helpful:
- Test Results Analyzer Plugin: this plugin lets you see the history of a particular test case (or test suite) at one glance (plus adds nice charts)
- Flaky Test Handler Plugin: this plugin has deeper support for failing tests (i.e. rerunning them). It's a bit restricted to maven and git though.
pitseeker
- 2,535
- 1
- 27
- 33
0
You label such tests not helpful. And your primary goal is to eliminate the "flakiness" of such tests - by identifying the root cause of the problem and either fixing production or test code; or both; or worst case by deleting or @Ignore'ing those tests.
Disclaimer of course: jenkins can't tell you about flaky testcases. How could it?!
If you assume that you have flaky testcases, you could spent some time to create a jenkins setup were all tests are run 5, 10, 50 times against the same build output; to then compare statistics. But this is nothing that would come for free - you will have to implement that yourself.
GhostCat
- 137,827
- 25
- 176
- 248