Study about the benefits unitTest

Question

Good afternoon. Is there a study about the benefits of automated(Unit and Integration) tests. I try to prove to my colleagues that it is useful and will bring profit to the company, but among the arguments I would like to refer to the results of research.

Answer 1

At one of my previous employers we have performed such a study ourselves. It was not published, however, but the results were clearly indicating that unit-testing was beneficial - in the sense that it more than paid for itself taking the whole development cycle into account. The people involved in the study, however, where trained and experienced in unit-testing and eager to continuously improve. Which means, they knew how to apply unit-testing in a sensible way (including dealing with dependencies etc.) and, maybe more important, they knew where unit-testing does not make sense. There were also no formal coverage goals, so there was no pressure to apply unit-testing to areas where it did not make sense.

When measuring the cost and the benefit, we tried our best to be neutral and consider all efforts involved, including time to write and run tests and analyze results, refactoring time for testability, training time, discussion time (asking colleagues how to unit-test some function), maintenance time of tests when the SUT changes etc.. On the benefits side we recorded the number of bugs that could be found earlier because of doing the unit-testing.

We saw, however, that it has a tremendous impact whether or not unit-testing is done right. That means, you can apply it in ways that will increase your development efforts without bringing the intended defect reduction. For example, in a different organizational unit, a development team was forced to apply unit-testing. However, they did their unit-testing after they had already heavily debugged the code. Not surprisingly, they found that with unit-testing they would not find as many bugs as expected. The importance of "doing it right" is also nicely discussed by Meszaros at http://xunitpatterns.com/Goals%20of%20Test%20Automation.html.

What our study consequently showed was, that unit-testing is beneficial if performed by people skilled and motivated to "doing it right", where doing it right includes "non-dogmatically". If performed by people who are only doing it because of being forced, possibly driven by formal coverage goals, you will likely find that unit-testing is not beneficial.