Using random numbers vs hardcoded values in unit tests

Question

When I write tests, I like to use random numbers to calculate things.

e.g.

 func init() {
   rand.Seed(time.Now().UnixNano())
 }

  func TestXYZ(t *testing.T) {
     amount := rand.Intn(100)
     cnt := 1 + rand.Intn(10)
     for i := 0; i < cnt; i++ {
       doSmth(amount)
     }
     //more stuff
  }

which of course has the disadvantage that

expected := calcExpected(amount, cnt)

in that the expected value for the test needs to be calculated from the random values.

If have received criticism for this approach:

It makes the test unnecessarily complex
Less reproduceable due to randomness

I think though that without randomness, I could actually:

Make up my results, e.g. the test only works for a specific value. Randomness proves my test is "robust"
Catch more edge cases (debatable as edge cases are usually specific, e.g. 0,1,-1)

Is it really that bad to use random numbers?

(I realize this is a bit of an opinion question, but I am very much interested in people's point of views, don't mind downvotes).

There's a difference between unit testing and fuzz testing. Each has value but it's important not to conflate the two. — Adrian, Jun 27 '19 at 22:13
This question seems basically opinion-based, but see [Testing: deterministic or non-deterministic?](https://softwareengineering.stackexchange.com/q/221632) on Software Engineering, or [What are the downsides using random values in Unit Testing?](https://stackoverflow.com/q/3441686) and [Random data in Unit Tests?](https://stackoverflow.com/q/32458) here. — dbc, Jun 27 '19 at 23:14

score 2 · Accepted Answer · answered Jun 27 '19 at 21:14

Your question is not specific to Go. This applies to any language and really any kind of unit testing.

the expected value for the test needs to be calculated from the random values.

That is the major problem. If you have even moderately complicated application logic when the logic changes in the application, you also have to change that same logic in your tests. You have to implement those changes TWICE.

Presumably each one is equally complex and each one is implemented somewhat differently because if you're just copy-pasting or reusing code from your app in your test to compute the expected value, then they're going to agree and the test is pointless.

Testing with fixed values in your unit tests keeps the tests simple and exercises the code.

Testing with random values is done, its called fuzzing. I am not an expert in fuzzing in tests. Testing with random values is one aspect of fuzzing but the nuance is in testing random values that are likely to find edge cases, catch bugs, exercise unused code branches, or uncover leaks.

Using random numbers vs hardcoded values in unit tests

1 Answers1