39

I've recently gotten into a team that heavily utilizes unit testing. Nobody can explain to me why this form of testing is so important, but they treat it like law.

I understand that the idea of automated testing is to prevent regression, but I don't see how that could be a problem in the first place. Modular, object-oriented, concise code that is well-commented doesn't have a problem with regression. If you build it right the first time, and design for the inevitable slough of feature adds that happen in the future, you'll never need tests.

And further, isn't that what graceful error handling and logging is supposed to accomplish? Why spend weeks hashing out assert statements and unit tests when you can just ensure that all your external dependencies double-check their availability first?

Am I being arrogant in coming to the conclusion that unit testing is a crutch for "bad" codebases which are flawed and built poorly?

This is a serious question. I can't find any good answers anywhere, and everyone I ask seems to think I'm being a troll if I question the purpose of automated testing.

EDIT: Thanks for the answers, I think I'm understanding it now. I see a few people voted to delete, but I'd like to thank the people that answered; it really did help!

pb2q
  • 58,613
  • 19
  • 146
  • 147
Knetic
  • 2,099
  • 1
  • 20
  • 34
  • 3
    I'm not sure anyone could respond better than [Uncle Bob](http://butunclebob.com/ArticleS.UncleBob.TheThreeRulesOfTdd). – David Robinson Aug 11 '12 at 20:23
  • Even if you build it right the first time, the world may still change causing the requirements to change. – Thorbjørn Ravn Andersen Aug 11 '12 at 23:34
  • 5
    "If you build it right the first time, and design for the inevitable slough of feature adds that happen in the future, you'll never need tests." Can you point out a single instance for a nontrivial system when this has happened? – thedayturns Aug 16 '12 at 00:46
  • BTW: preventing regressions has very little to do with unit testing. That's called **regression testing**. The only relation I can think of is that a regression test can be written in a form of a unit test. – Robert Koritnik Aug 21 '12 at 16:53

9 Answers9

42

No one is perfect - you make mistakes eventually. Unit testing is designed to catch and pinpoint the location of a mistake, in order to:

  • increase the confidence about the correctness of the code you write
  • increase the confidence about the correctness of refactors
  • make tracking down where a bug was introduced in the testing phase much simpler

Error handling and logging only helps when the bug is triggered; unit testing is what makes bugs be triggered in testing rather than production.

Consider the following...

You have a piece of software with 3 different parts, each of which has 2 different options.

     A      C      E
    / \    / \    / \
in-<   >--<   >--<   >-out
    \ /    \ /    \ /
     B      D      F

You could test this by manually putting in inputs and checking outputs - first you'd put in some inputs that triggered A,C,E; then you'd put in some that did A,C,F, and so on, until you covered everything through B,D,F.

But keep in mind that B, D, and F each have their own individual parameters and flows that need to be tested - we'll say there's maybe 10 variations for each. So that's at least 10*10*10 = 1000 different inputs you need to check, just for the A,C,E case. There's 8 different possible flows through these 6 components, so that's 8000 different combinations of inputs you need to check just to make sure you hit all of the different possible inputs.

On the other hand, you could unit test. If you clearly define the unit boundaries for each component, then you can write 10 unit tests for A, 10 for B, and so on, testing those boundaries. That gives you a total of 60 unit tests for the components, plus a handful (say 5 per flow, so 40) integration tests that make sure all of the components are tied together properly. That's a total of 100 tests which accomplish effectively the same coverage of functionality.

By using unit testing, you've reduced the amount of testing required to get an equivalent amount of coverage by a factor of about 80x! And that's for a relatively trivial system. Now consider more complex software where the number of components is almost certainly greater than 6, and the number of possible cases those components handle is almost certainly greater than 10. The savings you get from unit testing rather than just integration testing keep building.

Amber
  • 507,862
  • 82
  • 626
  • 550
  • Sure, bugs happen. It's a fact of life. But isn't catching them in pre-prod (where you're going to make sure that it works as anticipated anyway) using good logging and error handling, better than writing code that is not actually adding to its functionality or behavior? – Knetic Aug 11 '12 at 20:26
  • 7
    You say "catching them in pre-prod" as if that just magically happens with no effort required. Unit testing is part of the effort used to catch bugs in pre-prod. – Amber Aug 11 '12 at 20:27
  • 1
    @Knetic: how do you catch bugs in pre-prod, if you don't write any tests? – jalf Aug 11 '12 at 20:29
  • 1
    By using the software. Running it, looking at it from a user's point of view (or systems, if it's back-end), feeding it bad input, and trying edge cases. If it encounters errors, logs warnings, or anything unexpected, you've found a problem. – Knetic Aug 11 '12 at 20:40
  • Therein lies the rub, though - "using the software" doesn't guarantee that you've hit all of the edge cases. In fact, just using the software might make it *impossible* to hit all of the edge cases, because some of them might not be exposed to the user in the current version of the application and yet can become exposed later, indirectly. Furthermore, it's unlikely that just by using the software you'll be able to efficiently cover all of the possible combinations of interactions between the individual elements of your program. Even if you could, it would be highly inefficient. – Amber Aug 11 '12 at 20:42
  • @Knetic - I've seen the looks on the faces of developers that have automated tests on their code. A small startup at Google I/O last year had two developers and were demoing their product. They looked extremely cool, confident, calm, and collected, knowing that their app wasn't going to break while someone at the conference was playing around with it... Can't say the same for code written without tests. Those developers look nervous, stressed, and panicked. – jamesmortensen Aug 11 '12 at 20:46
  • 6
    @Knetic: for a complex system, even just testing all the "happy day" use cases (without going into error cases that you *can't* test that way) will occupy someone for many hours. Then he finds errors. They are fixed. To be certain the fixes have not broken anything, the many hours of testing have to be done *again*. And again, multiple times for any release. – Michael Borgwardt Aug 11 '12 at 20:58
  • @Amber like your part about `code coverage` Im glad my prof beat my head with it every day ... – Samy Vilar Aug 11 '12 at 21:10
  • 3
    @MichaelBorgwardt agree. The real pain point is that _any_ code change invalidates _all_ tests, and they need to be redone again _every single time_ – Thorbjørn Ravn Andersen Aug 11 '12 at 23:37
  • The problem is that if you don't do unit tests, then you need to hire QA to verify it. This becomes very costly when you release subsequent versions, because QA has to test it all over again. On the other hand, if you have automated tests, you don't need to spend additional man hours. – Rafal Rusin Sep 07 '12 at 19:14
23

Short answer: yes, you're arrogant. ;)

Assuming you truly are perfect, your code is not only correct and flawless when you write it, but it also takes into account all future requirements that will be placed upon it.

Now.... How do you know that your code is perfect and correct? You need to test it. If it hasn't been tested, you can't trust that it works.

It's not just about regressions (since that implies that the code used to work. What if it never worked? It was buggy when it was first written)

I understand that the idea of automated testing is to prevent regression, but i don't see how that could be a problem in the first place. Modular, object-oriented, concise code that is well-commented doesn't have a problem with regression.

Who told you that? That person ought to be flogged. Object-oriented code is just as error-prone as anything else. There's nothing magical about it, it's no silver bullet. At the end of the day, whenever you change a piece of code, there's a chance that you break something, somewhere. The chance might be larger or smaller depending on the code in question, but no matter how good the code, you can't be sure that you haven't introduced a regression unless you test it.

If you build it right the first time, and design for the inevitable slough of feature adds that happen in the future, you'll never need tests.

How do you build it right the first time, though? As I said above, to do so, you need to have tests, to show you that the code works. But more importantly, how do you "design for" features that will be added in the future? You don't even know what they are yet.

And further, isn't that what graceful error handling and logging is supposed to accomplish? Why spend weeks hashing out assert statements and unit tests when you can just ensure that all your external dependencies double-check their availability first?

No, not at all.

Your code should certainly handle error conditions and it should certainly log what you need logged.

But you still need to know that it does all this correctly. And you need to know that it handles the non-error conditions correctly too! It's great to know that "if the SQL server is unavailable, we show a nice error message to the user and exit". But what if it is available? Does your application work then?

For any nontrivial application, there are a lot of things that can go wrong. There's a lot of functionality, a lot of code, and a lot of different execution paths.

Trying to test it manually is never going to exercise all these code paths. It's never going to get around to testing every aspect of every feature in every context. And even if it did, that just tells you that "the code worked today". Will it work tomorrow? How can you know? Sure, your gut feeling might tell you that "the code I committed since then hasn't broken anything", but how do you know that? You need to test it again. And again. And again.

You ask if unit tests are a crutch for bad code bases. They're the opposite. They're the check-ups, the doctor visits, that prevent code bases from going bad. They don't just tell you whether or not your code works, but when it works, and more importantly, when it stops working. You don't think you're going to introduce any regressions? How sure are you? Can you afford to be wrong?

Dharman
  • 30,962
  • 25
  • 85
  • 135
jalf
  • 243,077
  • 51
  • 345
  • 550
  • 2
    The final point is really my main one, i should have written it better. What i mean is that, before deploying to the users, you're probably going to use a pre-production environment. In this environment, you can do 'dry runs' to ensure that your system is working as intended. And if it isn't, graceful error handling, error logging, and especially ensuring that (for instance) if that one SQL server isn't available, you don't charge in and make a query anyway (double-checking dependencies). Code that does that, especially in pre-prod, seems to have already done the job of unit tests. – Knetic Aug 11 '12 at 20:30
  • 3
    So you're going to look at every single possible use case -- of which there may be thousands -- manually? During each deployment? I've seen this in my work, and it sucks. It just takes up a lot of time that could be spent actually programming, and it never works. – jamesmortensen Aug 11 '12 at 20:41
  • 1
    @Knetic so you're basically saying "why let the computer do automatically what I could do manually"? The computer can run through thousands of tests in a second. You, the developer, would spend months to do the same manually. That's why automated tests are a good investment – jalf Aug 12 '12 at 09:40
  • But automated tests are still written by a person, who may not cover all conditions. Not to mention that methodical automated tests don't replicate the unexpected combinations of decisions made by a random user. – Oscar Apr 12 '22 at 12:09
8

When you first start writing your code it might seem like it is very simple and that automated tests aren't needed.

But over time, your code grows, the requirements change, and the team changes. The need for automated tests will also grow. Without automated tests, developers will be afraid to refactor the code - especially if they weren't the ones who wrote it. Even with carefully designed code adding new features can break existing features. But code isn't always carefully designed. In practice you may need to make compromises. For various reasons some of your code will not be as clean and maintainable as you might want. By the time you realize that you need automated tests, it might not be possible to add them because some of your classes could be difficult to test. To add the first automated test you may have to first rewrite or refactor large amounts of code.

If you have automated tests from the start, this will ensure that you design your code to be automaticaly testable.

Mark Byers
  • 811,555
  • 193
  • 1,581
  • 1,452
  • 2
    Manual tests can be done with any code. – Michael Borgwardt Aug 11 '12 at 20:21
  • 1
    @MichaelBorgwardt manual tests tend to suck at coverage (and the ones that don't are highly inefficient with regards to person-time). – Amber Aug 11 '12 at 20:22
  • 1
    @MichaelBorgwardt - "Manual tests" isn't testing, it's just [looking at output](http://stackoverflow.com/a/10859000/552792). (Note: I'm assuming by manual test you mean have a human use the software on a staging server and look at the results of each use case) – jamesmortensen Aug 11 '12 at 20:38
  • 1
    @jmort253: yes, that's what I mean, and OF COURSE it's testing. Sure, automated tests are overall much better (though not in every aspect), but claiming that anything else isn't testing is complete bullshit. – Michael Borgwardt Aug 11 '12 at 20:47
  • @MichaelBorgwardt - One might say the same thing about manual testing. It doesn't provide nearly the same confidence as something automated that covers thousands of use cases in just mere seconds. Sure, it might work for code you're never going to change. But if you have several people constantly making changes, I'm going to have a lot of reservations about putting my name behind it and making guarantees it's not going to break in production. – jamesmortensen Aug 11 '12 at 20:51
  • 1
    @MichaelBorgwardt: I did not mean to imply that manual testing is not testing. I have updated my answer to try to clear up this confusion. – Mark Byers Aug 11 '12 at 21:00
  • 1
    @jmort253: Sure, I know and agree with all that, but it's still testing and catches a large percentage of bugs, including some that no automated test will. – Michael Borgwardt Aug 11 '12 at 21:02
  • @MichaelBorgwardt - Fair enough. I can definitely agree with that. I'm not implying that this process has no place, just that it shouldn't replace automated testing. It should instead compliment it. – jamesmortensen Aug 11 '12 at 21:10
4

I ask the same thing to one of my professor, and long time mentor, that swears by it, one out the many that he has mentioned or have being mentioned here is this:

He said tests can be a great way to document your code, if you are interested to see how the code should behave look at the tests, it'll give you an idea of what its trying to do without having to figure out its complexities, and at the same time you can check if its actually doing that, so its a win win... :)

Though I have a feeling he just said that since he knew I don't like commenting code but prefer self documented source code, still kind of interesting point of view ;)

As a side note you should also take a look at coverage, when you do unit tests or design test suites you would want coverage to be as close to 100% as possible meaning that the tests have tested 100% of your code, including all the different paths it can take, this is quite challenging, and can make your test code several times larger than your source, though you can also automate your test code, a simple example would be testing a sql database, you can create a program that generates all the possible sql statements and test they are properly executed, I think sqlite3 has over 91 million lines of test, http://www.sqlite.org/testing.html very interesting... oh this something he also advocates.

Samy Vilar
  • 10,800
  • 2
  • 39
  • 34
  • "look at the tests, it'll give you an idea of what its trying to do" - Your prof. is a smart guy. This is of course the real point of unit tests, which none of the other answers really capture. Developers aren't any smarter when it comes to writing tests than they are when writing code. All of the other answers seem to assume, while the code may be bad, the tests will somehow magically be perfect. If a developer's understanding of the problem is wrong, the same bug will be in their tests as in their code. But the tests are a much clearer documentation of where the problem lies, and why. – Joe Stevens Mar 07 '21 at 07:17
4

I understand that the idea of automated testing is to prevent regression, but i don't see how that could be a problem in the first place.

In addition to the good advice given already, I'll add that regression is inevitable. Work in the industry for even a short time - well, long enough to go through multiple release cycles - and you realize that old bugs tend to pop up again, for lots of different reasons.

One particular reason: programmers tend to make the same kinds of simple mistakes over and over. For instance: when one of your devs is re-factoring that date handling code, he'll continue to make the incorrect assumption that e.g. February always has 29 days, or even worse, that all months are the same length.

Regression testing is your only defense against this sort of thing.

pb2q
  • 58,613
  • 19
  • 146
  • 147
4

Ok, so I was previously thinking a bit like you seem to do now, so I thought I would comment on some of your statements and just give an alternate point of view of you statements. Maybe this could shed some light on the different approach of coding that test-driven development really is.

I understand that the idea of automated testing is to prevent regression, but i don't see how that could be a problem in the first place.

Is a regression quite often not simply a modification done with an insufficient understanding of the workings of the code previously in place? If it is, how do you make sure that all developers document the functionality of their code in such a way that you automatically can detect if anyone at a later stage makes a destructive change? Could unit-testing perhaps help?

Modular, object-oriented, concise code that is well-commented doesn't have a problem with regression.

It might not have a problem with regression if you assume that the documentation is so well written that it is nearly impossible to misinterpret it and that each subsequent modification to said program is done in such a surgically exact way by a sufficiently skilled coder that it does in fact in no way alter the old behavior.

If you consider unit-testing as a way of documenting behavior, could it not be a equally or more effective way of communicating the functionality of the old code?

Testing should of course never preclude documentation, but each tool has its own pros and cons.

If you build it right the first time, and design for the inevitable slough of feature adds that happen in the future, you'll never need tests.

The cost and effort involved in building everything correct and flexible from the start in my opinion huge and often result in two things: An overly complex system and a slow development pace.

Ask yourself: what is the cost of flexibility? Are you minimizing risk by building several solutions to every problem and facilitating a run-time switch between them? Or are you building plugin systems allowing for patching behavior at run time? Both these strategies are horrendously expensive, both in development time as in adding complexity to the project.

If you just build what you need right now, and try to remain flexible enough as a team to quickly code any amendments as the need arises, how much time and effort would you save on every implementation?

If you have tests on existing functionality, will extending the functionality become easier?

And further, isn't that what graceful error handling and logging is supposed to
accomplish? Why spend weeks hashing out assert statements and unit tests when you can just ensure that all your external dependencies double-check their availability first?

Graceful error handling and logging might be good as a fallback, but fault recovery is never a replacement for correctness. I feel I cannot comment further since I am not exactly sure what you are referring to when you say "graceful error handling" and "double-checking availability", but this sound like recovery to me. Not having errors is much better than being able to recover from them.

Am i being arrogent in coming to the conclusion that unit testing is a crutch for "bad" codebases which are flawed and built poorly? This is a serious question. I can't find any good answers anywhere, and everyone i ask seems to think i'm being a troll if i question the purpose of automated testing.

You can certainly use unit testing as an excuse for avoiding documentation and good coding principles. This I have seen at many places. But it is also a great tool to add to your toolbox to increase the correctness, simplicity and flexibility of your product! Good luck with your new team. I believe that you can teach them a thing or two about good documentation and other principles which helps avoiding bugs, and they can probably bring some principles which will help you develop your product quicker and with less effort whilst avoiding exponential complexity growth.

Alexander Torstling
  • 18,552
  • 7
  • 62
  • 74
4

I have just two more bits to add:

  1. I find that the greatest benefit to unit testing is the subtle shift that happens when I have to write tests that are clients of my code. If it's hard to test, it's hard to use. I often make interface changes that end up making user's lives better.
  2. Writing unit tests is a form of documentation. It tells people "here's how to use (and misuse) my classes." Future clients should have lots of built in examples in how to use your code; unit tests give them.
duffymo
  • 305,152
  • 44
  • 369
  • 561
2

Modular, object-oriented, concise code that is well-commented doesn't have a problem with regression.

Even if it's well commented, the comments only describe the intended behavior. A function could sort a linear collection, but use bubblesort instead of quicksort. At that point, the comments become nothing better than a red herring (especially if you get a complaint that the sort takes forever, but you argue that, after an hour, it sorts properly). It could very well be the case that the original designer of the sort algorithm intended to use bubblesort on really small lists, then switch to quicksort on larger lists, but without tests to verify this, you have no way to check without digging into the code base.

Furthermore, functionality and specifications of a code base can change as quickly as an hour, depending on what you or your shop does. The documentation serves to be a major barrier to actual development. What's better would be to have a suite of test cases to ensure a lack of regressions.

If you build it right the first time, and design for the inevitable slough of feature adds that happen in the future, you'll never need tests.

Two words: agile development. Design is only "just enough", and adding more features than what's specified at the time violates YAGNI in a very big way. You'd wind up spending more time developing for the edge cases than developing for the main case.

And further, isn't that what graceful error handling and logging is supposed to accomplish? Why spend weeks hashing out assert statements and unit tests when you can just ensure that all your external dependencies double-check their availability first?

Error handling tells you that something went wrong. Unit tests can tell you where. You can only ensure that everything is as it should be with your codebase with proper tests.

Anyone can write obfuscated code and tell you that it's positively correct. Anyone inheriting that code base can't tell. Unit tests help mitigate this issue.

Am i being arrogent in coming to the conclusion that unit testing is a crutch for "bad" codebases which are flawed and built poorly?

A bit. Code bases don't have to be "bad" to need unit tests, they only need to change. (Changing code bases and code that doesn't need unit tests cannot exist in the same dimension.)

Another scenario which you haven't considered would be a code base that you inherit, and your clients claim that there is functionality missing or not working properly. It could be one or two pieces of it, but they have some really funky dependencies; you can't just take your favorite editor and go through it with a sledgehammer. In order to guarantee that you don't affect code that works for the time being, and wish to fix/implement new code, you would have to write regression tests for the modules you wish to change.

It's also the case that a bug gets more expensive to fix, the further up the chain it propagates. A bug found in development is less expensive to fix than a bug found in QA than a bug found in a release candidate than a bug found in a public release.

There's a few other lingering issues here, such as the cost of programming (maintenance costs are the brunt of it, not new development), as well as adding new features to legacy code, but for the most part, it is very important that unit tests get done. If you're working in a code base that doesn't have some sort of unit regression, you'll be spending too much time fixing bugs that tests could've covered.

Makoto
  • 104,088
  • 27
  • 192
  • 230
-1

100% coverage of unit testing allows your code to be executed from two different viewpoints. Usually your code is only executed when the program is run. Unit testing tests the flows in another more modular way. Think of double entry bookkeeping in accounting. (of which I have heard is the cornerstone to modern capitalism) Here every event appears twice in two different ledgers. At the end the balance should be zero. This form of book-keeping is doulbe checking itself all the time. This is how I think of unit testing.

Executing test first design I also find makes my code clearly and more management. By writing a test first I can think exactly about what classes should do.

RNJ
  • 15,272
  • 18
  • 86
  • 131