6

One caveat that some developers argue about TDD with Example-based tests is the possible lack of every valid input handling.

Let's take a simple example where those developers could argue:

Make a program that add two numbers.

Bob, a developer starts writing the first test:
Given 1 + 3, then result is 4.
Implementation: def add(x: Int, y: Int) = 4
Nice, it passes.

Now second test:
Given 2 + 2, then result is 4. Implementation still the same: def add(x: Int, y: Int) = 4

So other developers would come and tell him:

"Now you see Bob that Example-based testing is not enough with your poor 2 examples !
You didn't test every output and looking at your actual implementation, it will fail for other inputs whose result is distinct from 4!
Now, listen to us and start writing some property-based tests to cover all valid inputs.
So let's start with commutativity test, associativity etc.. that are specific to the addition: properties of the addition !"

But but but.... Bob's TDD practice is really bad!
Indeed, he wanted to apply triangulation but he wrote a test that already succeeded, since the implementation hadn't to be altered!

To lead to Triangulation, one should write a test that fails. And triangulation is one of the master key of the practice of TDD.
It allows the main step: refactoring since you should manage two paths that lead to 2 distinct results!

=> As long as the tests get specific, the code gets more generic thanks to refactoring.

So my question is:
If we practice strict TDD with good practice of triangulation, what is the real benefit of using Property-based testing, asserting invariants that are in 99% of cases already cover by a good TDD?
Indeed, supposing that developers have a good IQ and build implementation that makes sense ;)

My example is taken from those slides.

Mik378
  • 21,881
  • 15
  • 82
  • 180
  • 1
    You are supposing that the property being tested is simple enough that "a good IQ" will avoid the possibility that a corner case is missing. What happens, in your example, when the developer does not consider that someone may attempt to add the type's Max value to itself? – Angelo Genovese Jun 22 '15 at 18:59
  • While measuring the risk to have an edge bug like this (Max Value) VS implementing the solution more quickly using TDD habits, I would use the later. When this bug would happen, a simple additional test `addMaxNumberWithMaxNumber` would cover it, if forgotten. – Mik378 Jun 22 '15 at 19:00
  • 1
    I would see a benefit to practice property-based testing for use cases where there are too many distinct combination of paths to test (more than 10). But those kind of use cases are not really frequent in real enterprise project IMO. – Mik378 Jun 22 '15 at 19:11
  • The three way of moving forward when stuck in TDD is *Fake It*, *Obvious Implementation* and *Triangulation*. Triangulation should only be used if the other two fail. – zhon Dec 08 '16 at 03:58

2 Answers2

5

property based testing is very helpful when edge cases are hard to find or there is so many of them that programmer can easily miss one. i used it for example when i was implementing hirschberg's algorithm. there is no obvious way to divide algorithm into smaller, trivial, easily TDD-testable pieces. and it's hard to hand-craft input that cover all possible algorithm paths. the best way is to generate large number of different input to cover all the paths. and when automatic checking find the bug, add that specific input to your regression tests

piotrek
  • 13,982
  • 13
  • 79
  • 165
1

If you practice TDD, you know it is a way of thinking about and doing design rather than than being about testing.

TDD originated as incremental, mostly, state based unit testing. Then Interaction-style (ie., mockist-style, or London-style) TDD changed how we think about code design code.

Property-based TDD has the opportunity to change the way we design code as well. Instead of generating examples to push our design forward, we use properties. This results in fewer tests that are more maintainable with better test coverage.

On the other hand, it is harder and requires more thinking than example based TDD.

zhon
  • 1,610
  • 1
  • 22
  • 31