When to stop when trying to write meaningful tests using TDD?

Question

I'm struggling to find when to stop writting test cases using TDD.

Let's say a have to write a method that only accepts some strings, it can only accept the strings ['red', 'green', 'blue'], it is required and can't be empty.

I write the first failing test, make it green, and so on and so forth until I have the test cases:

it('should accept red input', () => { /*...*/ }
it('should accept green input', () => { /*...*/ }
it('should accept blue input', () => { /*...*/ }
it('should not accept null input', () => { /*...*/ }
it('should not accept empty input', () => { /*...*/ }

At this point everything is passing, now should I call it a day and go or should I go and add a test if it fails for Purple? Is it meaningful to add this test? If it is, I still can name other 10 colours to test, should I consider them all too?

This example is simple, but there are cases with Regexes that have infinite combinations that I know that could be a problem and I can't add all test cases I can think of for time constraints. These are worst, because I don't know when to stop writing test code, and find when enough is enough.

I undertand I can't get a concrete anwser for this, but I would like to hear from experience what works most of the time.

Answer 1

A great answer is post here: Is there such a thing as excessive unit testing?

In short you consider the probability of failure and weigh that against the cost of writing an automated test case.

In your case you can test one other color to see if it properly rejects it. But testing for all circumstances is unnecessary and impossible.

If you start receiving repetitive errors on a certain value, you might want to add it to the test. Otherwise boundary checks will do (Empty or None, correct behaviour, one failure)

Answer 2

Generally you want to test for input classes instead of specific inputs, unless you know already that a specific input will produce a specific situation you need to cover with the tests.

In your example I would break it down to four tests:

1. Should accept a color in expected input space, e.g. 'red'
2. Should not accept a color not in expected input space, e.g. 'purple'
3. Should not accept an empty input
4. Should not accept a null input

You're right that with parsing the input space is huge and covers more cases than you can realistically test. In that case I'd go for some common cases (the null/empty inputs, the clearly expected and unexpected inputs, etc.) and try to think of specific scenarios where an input might be miss-classified as (un-)expected. Maybe there are two colors 'red' and 'reddish' and you want to test that the function covers both or one, but not the other. The specific colors are not important in this case, just that one contains the other.

Answer 3

At this point everything is passing, now should I call it a day and go or should I go and add a test if it fails for Purple? Is it meaningful to add this test? If it is, I still can name other 10 colours to test, should I consider them all too?

One approach to improving your evaluation of your test subject without enumerating the entire world is property based testing. Scott Wlaschin wrote a good introduction to the technique.

Kent Beck is known for calling attention to duplication of data between the test and the implementation, and J. B. Rainsberger has suggested that, as part of your "remove duplication" work, the data tends to move toward the test. So what does that mean?

In your case, it sounds like you might be able to separate the test subject into two parts -- one part looks like a database (map, lookup table) of color names, the other part some logic that does something interesting given a database of color names.

For the latter part, you end up with tests like "given a database of [red, green, blue], does the test subject take the thats-not-a-color path when the input is purple; given a database of [red, green, blue, purple], does the test subject take the that-is-a-color path when the input is purple".

That leaves you with the database to worry about. What you want to be careful of there is the fact that simply testing that you have typed the same data in two different places isn't particularly valuable.

You might also consider this observation from Beck

I get paid for code that works, not for tests, so my philosophy is to test as little as possible to reach a given level of confidence

Remember, time invested in tests is time not invested elsewhere. Sometimes the right play is to ship, and then come back to the code later when you have learned something new about how it should work.