What's the best way to test this?

Question

I'm going through the EdgeCase Ruby Koans. In about_dice_project.rb, there's a test called "test_dice_values_should_change_between_rolls", which is straightforward:

  def test_dice_values_should_change_between_rolls
    dice = DiceSet.new

    dice.roll(5)
    first_time = dice.values

    dice.roll(5)
    second_time = dice.values

    assert_not_equal first_time, second_time,
      "Two rolls should not be equal"
  end

Except for this comment that appears there:

# THINK ABOUT IT:
#
# If the rolls are random, then it is possible (although not
# likely) that two consecutive rolls are equal.  What would be a
# better way to test this.

Which (obviously) got me thinking: what is the best way to reliably test something random like that (specifically, and generally)?

Answer 1

IMHO most answers so far have missed the point of the Koan question, with the exception of @Super_Dummy. Let me elaborate on my thinking...

Say that instead of dice, we were flipping coins. Add on another constraint of only using one coin in our set, and we have a minimum non-trivial set that can generate "random" results.

If we wanted to check that flipping the "coin set" [in this case a single coin] generated a different result each time, we would expect the values of each separate result to be the same 50% of the time, on a statistical basis. Running that unit test through n iterations for some large n will simply exercise the PRNG. It tells you nothing of substance about the actual equality or difference between the two results.

To put it another way, in this Koan we're not actually concerned with the values of each roll of the dice. We're really more concerned that the returned rolls are actually representations of different rolls. Checking that the returned values are different is only a first-order check.

Most of the time that will be sufficient - but very occasionally, randomness could cause your unit test to fail. That's not a Good Thing™.

If, in the case that two consecutive rolls return identical results, we should then check that the two results are actually represented by different objects. This would allow us to refactor the code in future [if that was needed], while being confident that the tests would still always catch any code that didn't behave correctly.

TL;DR?

def test_dice_values_should_change_between_rolls
  dice = DiceSet.new

  dice.roll(5)
  first_time = dice.values

  dice.roll(5)
  second_time = dice.values

  assert_not_equal [first_time, first_time.object_id],
    [second_time, second_time.object_id], "Two rolls should not be equal"

  # THINK ABOUT IT:
  #
  # If the rolls are random, then it is possible (although not
  # likely) that two consecutive rolls are equal.  What would be a
  # better way to test this.
end

Answer 2

I'd say the best way to test anything that involves randomness is statistically. Run your dice function in a loop a million times, tabulate the results, and then run some hypothesis tests on the results. A million samples should give you enough statistical power that almost any deviations from correct code will be noticed. You are looking to demonstrate two statistical properties:

1. The probability of each value is what you intended it to be.
2. All rolls are mutually independent events.

You can test whether the frequencies of the dice rolls are approximately correct using Pearson's Chi-square test. If you're using a good random nunber generator, such as the Mersenne Twister (which is the default in the standard lib for most modern languages, though not for C and C++), and you're not using any saved state from previous rolls other than the Mersenne Twister generator itself, then your rolls are for all practical purposes independent of one another.

As another example of statistical testing of random functions, when I ported the NumPy random number generators to the D programming language, my test for whether the port was correct was to use the Kolmogorov-Smirnov test to see whether the numbers generated matched the probability distributions they were supposed to match.

Answer 3

There is no way to write a state-based test for randomness. They are contradictory, since state-based tests proceed by giving known inputs and checking output. If your input (random seed) is unknown, there is no way to test.

Luckily, you don't really want to test the implementation of rand for Ruby, so you can just stub it out with an expectation using mocha.

def test_roll
  Kernel.expects(:rand).with(5).returns(1)
  Diceset.new.roll(5)
end

Answer 4

It seems like there are 2 separate units here. First, a random number generator. Second, a "dice" abstraction that uses the (P)RNG.

If you want to unit test the dice abstraction, then mock out the PRNG calls, and make sure it calls them, and returns an appropriate value for the input you give, etc.

The PRNG is probably part of your library/framework/OS and so I wouldn't bother testing it. Maybe you'll want an integration test to see if it returns reasonable values, but that's a whole 'nother problem.

Answer 5

Instead of comparing values, compare object_id:

    assert_not_equal first_time.object_id, second_time.object_id

This assumes that other tests will check for array of integers.

Answer 6

My solution was to allow a block to be passed to the roll function.

class DiceSet
  def roll(n)
    @values = (1..n).map { block_given? ? yield : rand(6) + 1 }
  end
end

I can then pass my own RNG into the tests like this.

dice = DiceSet.net
dice.roll(5) { 1 }
first_result = dice.values
dice.roll(5) { 2 }
second_result = dice.values
assert_not_equal first_result, second_result

I don't know if that's really better, but it does abstract out the calls to the RNG. And it doesn't change the standard functionality.

Answer 7

Just create new array each time roll method called. This way you can use

assert_not_same first_time, second_time,
"Two rolls should not be equal"

to test object_id equality. Yes, this test depends on implementation, but there is no way to test randomness. Other approach is to use mocks as floyd suggested.

Answer 8

IMHO, randomness should be tested with dependency injection.

Jon Skeet answered to the general answer of how to test randomness here

I suggest you treat your source of randomness (a random number generator or whatever) as a dependency. Then you can test it with known inputs by providing either a fake RNG or one with a known seed. That removes the randomness from the test, while keeping it in the real code.

Example code of in our case may look something like this:

class DependentDiceSet
  attr_accessor :values, :randomObject

  def initialize(randomObject)
    @randomObject = randomObject
  end

  def roll(count)
    @values = Array.new(count) { @randomObject.userRand(1...6) }
  end
end

class MyRandom
  def userRand(values)
    return 6
  end
end

class RubyRandom
  def userRand(values)
    rand(values)
  end
end

A user can inject any random behavior and test that the dice are rolled by that behavior. I implement ruby random behavior and another one that return always 6.

Usage:

randomDice = DependentDiceSet.new(RubyRandom.new)
sixDice = DependentDiceSet.new(MyRandom.new)

Answer 9

rand is deterministic and depends on its seed. Use srand with a given number before the first roll and srand with a different number before the second roll. That would prevent repeating the series.

srand(1)
dice.roll(5)
first_time = dice.values

srand(2)
dice.roll(5)
second_time = dice.values

assert_not_equal first_time, second_time,
  "Two rolls should not be equal"

Answer 10

It seems a bit silly, to me. Are you supposed to be testing that the (psuedo) random number generator is generating random numbers? That's futile and pointless. If anything, you could test that dice.roll calls to your PRNG.

Answer 11

I solved the problem using recursion:

def roll times, prev_roll=[]
    @values.clear
    1.upto times do |n|
       @values << rand(6) + 1
    end
    roll(times, prev_roll) if @values == prev_roll
end

And had to add a dup method to the test variable, so it doesn't pass the reference to my instance variable @values.

def test_dice_values_should_change_between_rolls
    dice = DiceSet.new

    dice.roll(5)
    first_time = dice.values.dup

    dice.roll(5, first_time)
    second_time = dice.values

    assert_not_equal first_time, second_time,
       "Two rolls should not be equal"

  end

Answer 12

The pragmatic approach is simply to test with a higher number of rolls. (The assumption being that this test is for two consecutive rolls of the same number).

likelihood of two 5 roll sets being the same => 6**5 => 1 in 7776

likelihood of two 30 roll sets being the same => 6**30 => 1 in 221073919720733357899776 (likelihood of hell freezing over)

This would be simple, performant and accurate [enough].

(We can't use object_id comparison since tests should be implementation agnostic and the implementation could be using the same array object by using Array#clear, or the object_id may have been reused, however unlikely)

Answer 13

i just created a new instance

def test_dice_values_should_change_between_rolls
    dice1 = DiceSet.new
    dice2 = DiceSet.new

    dice1.roll(5)
    first_time = dice1.values.dup

    dice2.roll(5, first_time)
    second_time = dice2.values

assert_not_equal first_time, second_time,
   "Two rolls should not be equal"

  end

Answer 14

I solved it by simply creating a new set of values for each dice anytime the 'roll' method is called:

def roll(n)     
    @numbers = []
    n.times do
      @numbers << rand(6)+1
    end
end

---