58

I am looking for rules like:

A test is not a unit-test if:

  • it communicates with a database
  • it cannot run in parallel with other tests
  • uses the "environment" like registry or file system

What else is there?

Peter Gfader
  • 7,673
  • 8
  • 55
  • 56
  • 12
    Roy Osherove’s definition of a unit test: only in memory, runs fast, is repeatable, does not touch any external resources – Peter Gfader Sep 04 '09 at 11:14
  • In addition to whats been said, check: This [question](http://stackoverflow.com/questions/61400/what-makes-a-good-unit-test). – Finglas Aug 10 '09 at 22:37

8 Answers8

70

See Michael Feathers' definition

A test is not a unit test if:

  • It talks to the database
  • It communicates across the network
  • It touches the file system
  • It can't run at the same time as any of your other unit tests
  • You have to do special things to your environment (such as editing config files) to run it.
Carl Manaster
  • 39,912
  • 17
  • 102
  • 155
  • 15
    Certainly not. Tests that span multiple layers (for example business logic, view, and controller) are also not considered unit tests. – Malte Clasen Aug 10 '09 at 22:37
  • 7
    I don't think lists of characteristics are a useful way of defining concepts. It reduces understanding to a bunch of rules and trivia to remember. While beginners naturally crave rules - myself included, and I am a beginner in unit testing - in the long run it's far more effective to focus on the concept, why it matters, and then experiment and think through how it fits into your own context. So when beginners ask for rules, experts who care should first and foremost explain the idea, then list some examples. :) – The Dag Dec 06 '12 at 19:32
  • 1
    The charactereristics of a Unit test depends of the unit of code which we are testing. If you write a DaoTest for your db inserts/selection, we are actually unit-testing the mappings (in case of ORM) or expected mappings with respect to table structure. We are not actually testing the DB or layer, we only test the unit of code which we have writted in DAO. So a generic statement it if it talks to the database is not valid. – DarkKnight Jul 19 '14 at 05:11
  • 2
    -1 This answer is just "dogma". A more useful definition would lead developers to the creation of good, *useful* automated tests, rather than just trying to fit arbitrary criteria. The point about touching the file system, in particular, encourages the developer to try and *mock* the file system away, which is usually quite hard to do in practice; also, IO code tends to use low-level APIs, so this code is heavily implementation-specific; it's much better to indeed let the test *touch* the (local) file system. – Rogério Jan 18 '15 at 22:57
  • 1
    @Rogério The point is that the unit tests are independent of anything else. What if the file changes locations? Mocking up data isn't all that awful. – sparkyShorts Apr 08 '15 at 22:44
  • @sparkyShorts Being independent of a file's location is not the same as "not touching the file system". And trying to make a unit test "independent of anything else" only leads to poor tests which abuse mocking/faking for isolation from dependencies; I have seen this happen countless times; personally, I think that unit tests are overrated, and that integration tests are usually much better. – Rogério Apr 09 '15 at 14:58
  • But if you use a "unit testing framework" to write it, and there is no better fit than that for the test you are writing, I wonder if maybe it should still be considered a unit test. I think as much code as possible should be as immutable and pure as possible, but when testing the other code that is not that way, I would still call it a unit test. If you have a class that is not immutable/pure, and you are trying to test it, if you use mocks for the dependencies, then you are not testing the real class, you are testing the class with modifications, turning the test into a "mock"ery. – still_dreaming_1 Dec 30 '22 at 03:54
36

A test is not a unit test if it is not testing a unit.

Seriously, that's all there is to it.

The concept of "unit" in unit testing is not well-defined, in fact, the best definition I have found so far, isn't actually a definition because it is circular: a unit in a unit test is the smallest possible thing that can be tested in isolation.

This gives you two checkpoints: is it tested in isolation? And is it the smallest possible thing?

Please note that both of these are context-dependent. What might be the smallest possible thing in one situation (say, an entire object) might in another situation just one small piece of one single method. And what counts as isolation in one situation might be in another (e.g. in a memory-managed language, you never run in isolation from the garbage collector, and most of the time that is irrelevant, but sometimes it might not be).

Jörg W Mittag
  • 363,080
  • 75
  • 446
  • 653
  • +1 : very nice. Certainly as useful as the [list Carl cites](http://stackoverflow.com/questions/1257560/when-is-a-test-not-a-unit-test/1257583#1257583) – Martin Ba May 25 '11 at 07:14
  • 1
    +1: It's this sort of understanding we should all be sharing with each other, rather than lists and rules - because (a) they are at best typical, not universal, and (b) understanding the *concept* is the only thing that will enable you to know when to break the rules. – The Dag Dec 06 '12 at 19:22
7

It has no asserts, and is not expecting an exception to be thrown.

pgb
  • 24,813
  • 12
  • 83
  • 113
  • +1: tricky one, I think your IDE will give you an error or at least a warning if you miss that. :-) – Peter Gfader Aug 10 '09 at 22:28
  • I don't think there's such a warning, at least with what I use (VS, NUnit) – R. Martinho Fernandes Aug 10 '09 at 22:30
  • 2
    It passes silently. What you can do in some IDEs is tweak the default unit test template to include an `Assert.Fail()` or equivalent as the last line of the test method. This way, it will "fail by default" – pgb Aug 10 '09 at 22:32
  • 5
    Not 100% agree. You could have tests which just test that some code runs without throwing an exception. It is not ideal, but sometimes more than nothing at all. You could say that there is always an implicit assert that the unit under test does not throw an exception. – Stefan Steinegger Nov 22 '10 at 08:59
  • You could have Unit Test without an Assert. For an example in MSTest, you would write a Unit Test to check whether the correct exception being thrown by the system under test. – Spock Apr 21 '11 at 13:01
  • Just changed it a little bit, to include the case of expecting exceptions. – pgb Apr 22 '11 at 12:40
  • Isn't that an excessively language-specific criterion? – smci Jul 19 '11 at 10:16
  • Counter-example, even when appended to the OP's criteria: Automated tests that validate a grep or xslt program, fed through the entry point, using redirected IO to input data and to assert against. These are most certainly not unit tests by many people's definition, even if they use a unit testing framework. – Merlyn Morgan-Graham Apr 05 '12 at 07:38
  • @Raj If you're testing that a particular exception type is thrown, then that is what you are asserting. So then the test does contain an assertion, it just doesn't invoke any method on the Assert class. But 99% of the time it'd be better to do so, e.g. Assert.AreEqual(expectedExceptionType, ex.GetType()); – The Dag Dec 06 '12 at 19:25
  • @TheDag yep make sense, I first thought "no asserts" means, not using the Assert class. But that's not the case. – Spock Dec 11 '12 at 10:14
  • @pgb, where did this definition come from? What do asserts and exceptions have to do with the notion of a unit? What do you call an xunit test that has an assert or expects an exception if it fits Michael Feathers' and Roy Osherove's definitions of a unit test? – aro_tech Dec 24 '15 at 07:56
7

Difficult one...

For me a unit test verifies one specific piece of logic in isolation. Meaning, I take some logic, extract it from the rest (if necessary by mocking dependencies) and test just that logic - a unit (of the whole) - by exploring different kind of possible control flows.

But on the other side...can we always 100% say correct or incorrect?? Not to become philosophical, but - as also Michael says in his post:

Tests that do these things aren't bad. Often they are worth writing, and they can be written in a unit test harness. However, it is important to be able to separate them from true unit tests so that we can keep a set of tests that we can run fast whenever we make our changes.

So why shouldn't I write a unit test that verifies the logic of parsing for instance an xls file by accessing some dummy file from the file system in my test folder (like MS tests allow with the DeploymentItem)?

Of course - as mentioned - we should separate these kind of tests from the others (maybe in a separate test suite in JUnit). But I think one should also write those tests if he feels comfortable in having them there...clearly then always again remembering that a unit test should just test a piece in isolation.

What is most important in my eyes is that these tests run fast and don't take too long s.t. they can be run repeatedly and very often.

Community
  • 1
  • 1
Juri
  • 32,424
  • 20
  • 102
  • 136
  • Nit-pick: In your xls parsing example it would be ideal to feed the input via redirected IO instead of from the file system and to output to stdio if the program allowed it. Or some sort of memory-based file system. You could consider this truly isolated, and it would allow running tests in parallel. Test isolation isn't the only criteria for "true" unit tests, and it also gives major benefits in higher level tests. – Merlyn Morgan-Graham Apr 05 '12 at 07:42
  • 1
    There's not only the issue of speed... You should not need to do any configuration work like setting up a RDBMS to run unit tests. Like, checkout fresh codebase from repo -> run tests -> all green. – hijarian Nov 15 '13 at 11:21
6

A test is not an Unit Test when:

  • it tests more than one thing at once (i.e. it tests how two things work together) - then it is an integration test

Checklist for good unit tests:

  • they are automated
  • they are repeatable
  • they are easy to implement
  • they remain for future use, once written
  • they can be run by anyone
  • they can be run by the push of a button
  • they run quickly

Some more best practices (in no particular order of importance):

  • tests should be separated from integration tests (which are slower), so that they can be run fast as frequently as possible
  • they should not comprise too much logic (preferably, no control structures)
  • every test should test only one thing (thus, they should contain only one assert)
  • the expected values used in assertions should be hard-coded and not computed at test run-time
  • external dependencies (filesystem, time, memory etc.) should be replaced by stubs
  • test should recreate the initial state at test shutdown
  • in assertions, it is better to use a "contains..." policy, rather than "is strictly equal..." policy (i.e. we expect certain values in a collection, certain characters in a string etc.)

This is part of the knowledge I have extracted from Roy Osherove's book - The Art of Unit Testing

Peter Gfader
  • 7,673
  • 8
  • 55
  • 56
Ruxandra T.
  • 521
  • 1
  • 6
  • 10
  • 1
    I think "external dependencies (filesystem, time, memory etc.) should be replaced by stubs" is not just a best practice ;-) – Peter Gfader Sep 06 '11 at 06:24
  • 1
    When using mock objects you can follow the "contains" policy by checking that your intended methods were called, but ignoring that other methods were called. Strict mocks go against this (they throw assertions when unexpected methods are called), and make for more brittle tests. Of course if the interface you are mocking makes it important to call a method only x times, or to call certain methods in a specific order, then you may want to explicitly assert that. That assertion would be placed in its own separate test. – Merlyn Morgan-Graham Apr 05 '12 at 07:48
2

Implementing a test across multiple possibly failing units would not be a unit test.

Vince
  • 7,608
  • 3
  • 41
  • 46
1

Intricate question.

Say I am to program some business logic and all business logic needs to get to the data via some form of DAL.

Say that for the purposes of testing, I mock the DAL units (by creating "mockingbirds").

But those mockingbirds are of course, additional units in their own right. So even when using mocks, it might seem like I'm still bound to violate the idea of "no other units involved" when I want to unit-test my business logic module.

Of course, it is generally known that "creating mockingbirds for the DAL" can invalidate your very test itself on the count that your mockingbird deviates in some particular aspect from the DAL.

Conclusion : it is outright impossible to do "genuine unit-tests" on business modules that depend in any way on any kind of DAL, question mark ?

Corrolary : the only thing that can possible be ("genuinely" !) unit-tested is the DAL itself, question mark ?

Corrolary of the corrolary : given that the "DAL" is usually either an ORM or the very DML of some DBMS, and given that those products are usually bought as being "proven technology", what is the added value of doing any unit tests what so ever, question mark ?

  • interesting thoughts :-) .... I wouldn't say this: "it is outright impossible to do 'genuine unit-tests' on business modules .... and I wouldn't say that the DAL can be unit-tested.... – Peter Gfader Aug 11 '09 at 11:05
  • You must always assume (i.e. make sure) that you mock an interface to give the results you expect from an implementation, so that there is a correlation between mocks and real objects. Therefore, I must disagree with your corrolary: Sure, it might be hard to mock the DAL exactly the way it works, especially if the DAL is an ORM tool that you have not written yourself. However, by correctly mocking it you can still test the business modules, and by mocking the interface of your business modules, you can easily test anything "later" in the chain. – Tomas Aschan Feb 15 '11 at 23:41
  • I am working on a DAL layer too and the worst part is unit testing takes more time than coding. THere are some things you can test, others which you simply cant unless you assume data. There are some tests that pass intermittenly(for eg IO realted stuff can easily give unexpected results). Another problem is that I cannot run all test cases together. UTCs depend on the state. – DPD Mar 16 '11 at 05:27
0

After whether a test is a unit test or not is settled the next question is, is it a good unit test?

Community
  • 1
  • 1
Ates Goral
  • 137,716
  • 26
  • 137
  • 190