9

Using FsCheck, the F# version of the Haskell QuickCheck test library, to generate tests from C#, I found that the random string generator does not generate the null string.

using FsCheck.Fluent;
Spec.ForAny<string>(s => s != null).QuickCheck(); // always pass

Furthermore, there seems not to handle null strings by design, but I have not managed to pin it down from the documentation. For example, just picking between two strings, one of them null, won't work:

var strings = Any.ValueIn<string>(null, "non-null string");
Spec.For(strings, s => true).QuickCheck(); // throws null ref exception

And strings seem to be a special case, because it handles custom-made objects such as

class Thing {}

when mixed with null values:

var objects = Any.ValueIn(null, new Thing());
Spec.For(objects, s => true).QuickCheck(); // pass
Alapago
  • 368
  • 2
  • 11

2 Answers2

5

I tried to dig a bit into this and it appears that you have discovered a bug in FsCheck.

It appears that the problem is in file Arbitrary.fs and is really only string-related. I had to replace this, where they call ToCharArray on the string

    static member String() = 
        { new Arbitrary<string>() with
            override x.Generator = Gen.map (fun chars -> new String(List.toArray chars)) generate
            override x.Shrinker s = s.ToCharArray() |> Array.toList |> shrink |> Seq.map (fun chars -> new String(List.toArray chars))
        }

with this

    static member String() = 
        { new Arbitrary<string>() with
            override x.Generator = Gen.map (fun chars -> new String(List.toArray chars)) generate
            override x.Shrinker s = 
                match s with
                    | null  -> seq {yield null;}
                    | _ -> s.ToCharArray() |> Array.toList |> shrink |> Seq.map (fun chars -> new String(List.toArray chars))
        }

You may want to raise this with fscheck developers here and also check if my fix works well - there is probably a better way to implement it, but it would be simpler for someone, who already knows the code.

Tomas Pastircak
  • 2,867
  • 16
  • 28
  • If it is a bug then it might cause trouble if fixed. So they might upgrade it into a "feature". – Alapago Apr 23 '14 at 11:18
  • 1
    @user2046431 I strongly doubt it, as this case will fail for and only for null `string`. And even if they decide to upgrade it to a feature, they may at least want to track it somewhere on their site. – Tomas Pastircak Apr 23 '14 at 11:21
  • Agreed that this should go to their issue tracker, and that fixing the shrinker so that it does not choke on nulls wouldn't break anything. I was thinking of the "bug" of not generating nulls by default. – Alapago Apr 23 '14 at 12:02
  • 1
    Why don't you submit a pull request. Certainly the shrinker fix I'd accept. I'm not so sure about generating nulls; FsCheck's primary target historically has been F# and generally nulls aren't much of a problem there. – Kurt Schelfthout Apr 24 '14 at 06:44
  • Note that in FsCheck 2.x nulls are being generated and shrunk for string correctly. – Kurt Schelfthout Oct 13 '15 at 07:42
1

For FsCheck 1.x, I found a solution which involves modifying the default random string generator:

public class MyArbitraries
{
    public static Arbitrary<string> String()
    {
        var nulls = Any.Value<string>(null);
        var nonnulls = Arb.Default.String().Generator;
        return Any.GeneratorIn(nulls, nonnulls).ToArbitrary;
    }
}

and then initialise it with:

DefaultArbitraries.Add<MyArbitraries>();

Then the test in the question fails as intended:

Spec.ForAny<string>(s => s != null).QuickCheck() // now fails, which is good

This will generate around 50% nulls and 50% random strings, the weights can be adjusted:

Spec.ForAny<string>(s => true)
    .Classify(s => s==null, "null")
    .Classify(s => s!=null, "not null")
    .QuickCheck(); // displays percentages

However, effectively overriding the default string generator might not be a good idea if the decision of not including the null value by default was intentional and not a bug in the library. And, if it were a bug, it would distort the distributions when fixed.

Alapago
  • 368
  • 2
  • 11