15

I got the impression that some problems are just to hard to unit test. And even if you do it, often such tests provide little value.

What code should not be unit tested, apart from getters and setters?

(might be similar to this question)

Community
  • 1
  • 1
arturh
  • 6,056
  • 4
  • 39
  • 48
  • You should remember that unit testing is not just about finding bugs during development. They're also very useful when you need to test for regression. – Fernando Jul 05 '09 at 16:25

6 Answers6

15

My general approach is "if this code is not worth testing, why is it worth having in the first place"? If I'm using a language which forces me to have a lot of uselessly repetitive boilerplate, then maybe I don't need to test those parts if the language's compiler can just check them; but I normally use languages where code I write is actually meaningful;-).

Can you given an example of problem that's too hard to unit-test? I've heard this used as an excuse to avoid testing error-recovery and diagnostic code that's only triggered by rare and very unlikely circumstances, but every time this has come up I've argued that, on the contrary, that code is the one most needing unit tests, because it's not going to get exercised in the integration tests and normal use (e.g. at QA stage).

Dependency Injection lets you use a fake or mock object to stand for (whatever "should never cause this error but we're covering for it anyway" -- network, database, power control interface, etc), and your fake or mock easily can and definitely should cause fake errors of all kinds so you can thoroughly check that error-recovery and diagnostic code.

Maybe it depends on what kind of apps you write -- for the last few years I've been mostly in cluster-management software, where everything that can go wrong will, lots of things that can't possibly go wrong will anyway, and uptime and fast recovery are crucial. In that field nobody would ever dare argue against a belt-and-suspenders approach (if they did the reliability engineers would be after them with cudgels;-).

But I've recently switched to Business Intelligence and I've noticed the approach translates well, too: if the numbers my code is producing (maybe to show as a nice graph to business decision makers, etc) are worth producing at all, they'd better be accurate, which means (among other things) that the code producing them needs to be tested just as thoroughly and carefully as that which monitors a network or a power supply system!-)

Alex Martelli
  • 854,459
  • 170
  • 1,222
  • 1,395
  • 1
    @Alex: What if the boilerplate code is generated for you? Then you didn't have to write it, but it may not be worth unit testing, if you can assume the code generator did it correctly. – John Saunders Jul 05 '09 at 16:35
  • 1
    While I feel unit testing is useful, developer time is a scarce commodity and testing "everything" is prohibitively time consuming. I'd only say Frameworks should get anywhere near 100% (ignoring boilerplate) code coverage. – Kevin Montrose Jul 05 '09 at 16:39
  • The boilerplate may be generated for you (I'm thinking of code templates via Eclipse) but the tests will confirm a) it's been generated correctly b) it hasn't changed. If you can change how the boilerplate code is generated, there's room for error, and who's to say that someone doesn't change code not created by team members. Is this a little anal ? Perhaps. But I would rather put the effort in. – Brian Agnew Jul 05 '09 at 16:48
  • @John and @Brian, I agree that code generators may need a different mindset -- sometimes you can think of whatever input you write to give to the code generator as a higher-level language worth testing... but the code generator may not give you decent tools to do that (e.g. enabling dependency injection into the generated code to let you mock & fake things), in which case a reasonable compromise might be to only thoroughly test the "visible API" of that generated code (be it a class, procedures, SQL queries, whatever). – Alex Martelli Jul 05 '09 at 17:10
  • 2
    @Kevin, modern tools and approaches, and good process, combine to lower the costs of strong testing -- while the cost of bugs that are discovered later than feasible can still be very high. So you still need to ask: if this code is not worth testing, why was it worth writing? If a system must have "six nines" uptime, how can it afford to have less than "six nines" coverage in testing -- whether it's a framework or not?-) Maybe games & the like are different (i.e. maybe bugs are very cheap there, while time to market is crucial) -- I don't know, not being a game developer myself. – Alex Martelli Jul 05 '09 at 17:22
6

You shouldn't write unit tests for other people's code (such as a framework you are using). You should only write tests for your code. Mock out dependencies on other people's code so that you only need to write tests for yours.

tvanfosson
  • 524,688
  • 99
  • 697
  • 795
  • 2
    Usually true, although a few tests to check any undocumented (or thinly documented) assumptions you have about your external dependencies can be extremely helpful sometimes. – Doug McClean Jul 05 '09 at 16:35
  • I'm not suggesting that you omit integration tests, but I make a distinction between unit tests, which are run frequently and quickly, and integration tests, which are run less often and may need some extensive environment setup and require more time. – tvanfosson Jul 05 '09 at 16:42
5

This is a question of cost and benefit, the closer you try to get to 100% the more expensive it will be.

There is also the UI layer, if this is in a technology that is difficult to test, you could program this layer such that it is as thin as possible, and then only test it manually.

Depending upon your situation you could drop testing pass-through layers and generated code.

Note that it is not just a question of code coverage but how you test, it may be better to have many tests on a limited part of the code and a lower code coverage.

Shiraz Bhaiji
  • 64,065
  • 34
  • 143
  • 252
3

On my current project I'm doing automated testing, of features and of system functionality, but no unit testing at all: Should one test internal implementation, or only test public behaviour?

Some people talk as if the only alternative to unit-testing is ad-hoc manual testing; but many of the benefits (e.g. regression testing) come from testing being automated, and not necessarily from it's being at the unit level.

Community
  • 1
  • 1
ChrisW
  • 54,973
  • 13
  • 116
  • 224
2

You don't need to test the language constructs, but outside of that, there's really not anything that "shouldn't" be unit tested.

If there are cases where you've already got the design, and a good reason for it to exist, and it's not a mission critical part of the application, such as a minor user interface feature, then a case can be made that it's not necessarily worth fighting to produce a unit test for. But that's not necessarily the same as "shouldn't" be unit tested.

Gerald
  • 23,011
  • 10
  • 73
  • 102
0

Automated unit tests can't be run on graphics code, since the computer cannot decide if what's being drawn to the screen is actually correct or not! Although you can write a manual unit test in that case, of course

Martin
  • 12,469
  • 13
  • 64
  • 128
  • 1
    arturh asked what *should* not be tested, not what *can* not be tested :) – n3rd Jul 05 '09 at 16:23
  • 2
    Oh it's definitely possible to test graphics code :) Tools like AutomatedQA have screenshot tests where a pixel by pixel comparison can be done between screenshots. It's definitely not worth it though. – Praveen Angyan Jul 05 '09 at 16:29
  • @Praveen - I would say it's worthwhile checking graphics code. Just recently I worked on a project rendering complex charts to .pngs for delivery to customers. The ability to compare a generated .png's checksum vs. a reference is MUCH more reliable than a visual inspection, and trying to remember what's supposed to be correct. – Brian Agnew Jul 05 '09 at 16:50
  • I phrased that a little misleadingly, I meant realtime graphics (i.e. games), which would be rather hard to test! – Martin Jul 05 '09 at 17:04
  • @Martin - yes. I'm a little stumped by that too! – Brian Agnew Jul 05 '09 at 17:15