Testcases for NUnit 3 tests in F#

Question

I am trying to set up a test suite for an F# project using NUnit. It seems that especially when testing things like parsers and type checkers one typically has a list of valid input data and a list of invalid data. The tests itself are practically identical, so I am looking for a clever way to avoid writing a test function for each and every data item and instead seperate the test function from the data. Apparently, there seems to be something called test cases for that but I am having a hard time to find comprehensive documentation for the usage of NUnit 3 with F# in general and specifically a best practice example for my scenario.

Any pointers and hints are greately appreaciated!

Answer 1

This is an updated answer for NUnit 3.x since my original answer showed a NUnit 2.x example.

The examples below are not meant to be comprehensive, but enough to get you over the threshold and up an running. The things of note are that the test functions are written using argument list instead of currying. Also there are several ways to generate test data using NUnit 3.x attributes, e.g. Pairwise, but sadly none of the attributes available know how to generate test data for discriminated unions.

Also FSUnit is not needed and I didn't try to make it work as the difference between NUnint 2.x and 3.x are so dramatic that I was happy just to get the following examples working. Maybe in the future I will update this answer.

namespace NUnit3Demo

open NUnit.Framework

module MyTest = 
    // ----------------------------------------------------------------------

    [<Pairwise>]
    let pairWiseTest([<Values("a", "b", "c")>] (a : string), [<Values("+", "-")>] (b :string), [<Values("x", "y")>] (c : string))
        = printfn "%A %A %A" a b c

    // ----------------------------------------------------------------------

    let divideCases1 =
        [
            12, 3, 4
            12, 2, 6
            12, 4, 3
        ] |> List.map (fun (q, n, d) -> TestCaseData(q,n,d))

    [<TestCaseSource("divideCases1")>]
    let caseSourceTest1(q:int, n:int, d:int) =
        Assert.AreEqual( q, n / d )

    // ----------------------------------------------------------------------

    let divideCases2 =
        seq {
            yield (12, 3, 4)
            yield (12, 2, 6)
            yield (12, 4, 3)
        }

    [<TestCaseSource("divideCases2")>]
    let caseSourceTest2(q:int, n:int, d:int) =
        Assert.AreEqual( q, n / d )

    // ----------------------------------------------------------------------

    [<TestCase(12,3,4)>]
    [<TestCase(12,2,6)>]
    [<TestCase(12,4,3)>]
    let testCaseTest(q:int, n:int, d:int) =
        Assert.AreEqual( q, n / d )

    // ----------------------------------------------------------------------

    let evenNumbers : int [] = [| 2; 4; 6; 8 |]

    [<TestCaseSource("evenNumbers")>]
    let caseSourceTest3 (num : int) =
        Assert.IsTrue(num % 2 = 0)

Leaving Original Answer since it was noted in other answer by OP.

The following example was written 3 years ago using NUnit 2.x so it is a bit dated but should give you an ideal.

You create an array of the test data, then index into the array to pull out the test values and expected results. The nice thing about this is that you don't wind up writing lots of individual test for a function.

This comes from a project some of us did years ago.

open NUnit.Framework
open FsUnit

let private filterValues : (int list * int list)[] = [| 
    (
        // idx 0
        // lib.filter.001
        [],
        []
    ); 
    (
        // idx 1
        // lib.filter.002
        [-2],
        [-2]
    );
    (
        // idx 2
        // lib.filter.003
        [-1],
        []
    );
    (
        // idx 3
        // lib.filter.004
        [0],
        [0]
    );
    (
        // idx 4
        // lib.filter.005
        [1],
        []
    );
    (
        // idx 5
        // lib.filter.006
        [1; 2],
        [2]
    );
    (
        // idx 6
        // lib.filter.007
        [1; 3],
        []
    );
    (
        // idx 7
        // lib.filter.008
        [2; 3],
        [2]
    );
    (
        // idx 8
        // lib.filter.009
        [1; 2; 3],
        [2]
    );
    (
        // idx 9
        // lib.filter.010
        [2; 3; 4],
        [2; 4]
    );
    |]

[<Test>]
[<TestCase(0, TestName = "lib.filter.01")>]
[<TestCase(1, TestName = "lib.filter.02")>]
[<TestCase(2, TestName = "lib.filter.03")>]
[<TestCase(3, TestName = "lib.filter.04")>]
[<TestCase(4, TestName = "lib.filter.05")>]
[<TestCase(5, TestName = "lib.filter.06")>]
[<TestCase(6, TestName = "lib.filter.07")>]
[<TestCase(7, TestName = "lib.filter.08")>]
[<TestCase(8, TestName = "lib.filter.09")>]
[<TestCase(9, TestName = "lib.filter.10")>]
let ``List filter`` idx = 
    let (list, _) = filterValues.[idx]
    let (_, result) = filterValues.[idx]
    List.filter (fun x -> x % 2 = 0) list 
    |> should equal result
    filter (fun x -> x % 2 = 0) list 
    |> should equal result

IIRC the problem with using NUnit with F# is to remember to use <> in the right location.

Answer 2

In NUnit3 there is TestCaseSource and TestCaseData and for the best practices part I added FsUnit:

namespace NUnit3Demo

open NUnit.Framework
open FsUnit

[<TestFixture>]
module MyTest = 

    let methodToBeTested s = 
        if String.length s > 3 then failwith "Something's wrong"
        else String.length s

    let validData =
        [
            TestCaseData("   ").Returns(3)
            TestCaseData("").Returns(0)
            TestCaseData("a").Returns(1)
        ]

    let invalidData =
        [
            "    "
            "abcd"
            "whatever"
        ]

    let otherInvalidData =
        [
            "just"
            "because"
        ]

    [<TestCaseSource("invalidData");
      TestCaseSource("otherInvalidData")>]
    let ``More than 3 characters throws`` s = 
        (fun () -> methodToBeTested s |> ignore)
        |> should throw typeof<System.Exception>

    [<TestCaseSource("validData")>]
    let ``Less than 4 characters returns length`` s = 
        methodToBeTested s

Note that TestCaseData can take and return arbitrary objects (obviously they should match the test signatures). Also, the data can be written even nicer:

let validData =
    [
        "   ", 3
        "",    0
        "a",   1
    ] |> List.map (fun (d, r) -> TestCaseData(d).Returns r)

Answer 3

By the end of the day, I realised I should not have used an array in the first place!

I finally understood how the TestCase mechanism is supposed to work: it simply passes the annotation's contents as a parameter to a function that now is not unit->unit anymore but (in my case) string->unit. Thus all my data items are now pasted into individual TestCase annotations, and the array is gone. Surely this might look a bit odd, to have TestCase annotations whose contents span many lines of code but the array was not beautiful either, so be it.

Unfortunately, my solution is not universally applicable, e.g. would not work for Guy Coder's code above. The reason is indicated here: https://stackoverflow.com/a/28015585/2289899 where they say:

CLI has a restriction regarding kinds of attribute parameters:

• primitive: bool, int, float, etc
• enums
• strings
• type references: System.Type
• 'kinda objects': boxed (if needed) representation of types from above
• one dimensional array of one of types from above (ie. no nested arrays allowed)

So we can conclude at this point that you can not use tuple as an attribute parameter's type.