fscheck doesn't generate random enough data

Question

I'm playing with FsCheck so I have this implementation:

let add a b = 
    if a > 100
    then failwith "nasty bug"
    else a + b

...and this FsCheck based test:

fun (a:int) -> (add a 0) = a
|> Check.QuickThrowOnFailure

and the test never fails. My guess is that the 100 values produced by the random generator are never bigger than 100.

Shouldn't the values be more "random"?

score 11 · Accepted Answer · answered Nov 14 '16 at 15:09

When you use Check.QuickThrowOnFailure, it uses the configuration Config.QuickThrowOnFailure, which has these values:

> Config.QuickThrowOnFailure;;
val it : Config =
  {MaxTest = 100;
   MaxFail = 1000;
   Replay = null;
   Name = "";
   StartSize = 1;
   EndSize = 100;
   QuietOnSuccess = false;
   Every = <fun:get_Quick@342>;
   EveryShrink = <fun:get_Quick@343-1>;
   Arbitrary = [];
   Runner = <StartupCode$FsCheck>.$Runner+get_throwingRunner@355;}

The important values to consider here are StartSize, but particularly EndSize. Some of the generators in FsCheck uses the size context to determine the size or range of values it generates.

If you change the EndSize to e.g. 1,000 you can make your test fail:

> Check.One({Config.QuickThrowOnFailure with EndSize = 1000}, fun (a:int) -> (add a 0) = a);;
System.Exception: Falsifiable, after 15 tests (0 shrinks) (StdGen (1912816373,296229213)):
Original:
101
with exception:
> System.Exception: nasty bug
   at FSI_0040.add(Int32 a, Int32 b)
   at FSI_0055.it@69-6.Invoke(Int32 a)
   at FsCheck.Testable.evaluate[a,b](FSharpFunc`2 body, a a) in C:\Users\Kurt\Projects\FsCheck\FsCheck\src\FsCheck\Testable.fs:line 161

   at <StartupCode$FsCheck>.$Runner.get_throwingRunner@365-1.Invoke(String message) in C:\Users\Kurt\Projects\FsCheck\FsCheck\src\FsCheck\Runner.fs:line 365
   at <StartupCode$FsCheck>.$Runner.get_throwingRunner@355.FsCheck-IRunner-OnFinished(String , TestResult ) in C:\Users\Kurt\Projects\FsCheck\FsCheck\src\FsCheck\Runner.fs:line 365
   at FsCheck.Runner.check[a](Config config, a p) in C:\Users\Kurt\Projects\FsCheck\FsCheck\src\FsCheck\Runner.fs:line 275
   at <StartupCode$FSI_0055>.$FSI_0055.main@()
Stopped due to error

fscheck doesn't generate random enough data

1 Answers1

Linked