2

First, let me present the test setup in F# (using FsUnit on top of NUnit):

type SimpleRecord = { A: int; B: int } 
                    override x.ToString() = x.A.ToString()

[<TestFixture>]
type ``Simple Test Cases``() =     
    static member SimpleDataSource =
        [|
            [|{ A = 1; B = 2} ,3|]
        |]

    [<TestCaseSource("SimpleDataSource")>]
    member x.``SimpleTest`` (testData: SimpleRecord * int) =
        let data, expected = testData
        data.A + data.B
        |> should equal expected

This test will run and pass as expected. However, changing the ToString override to include a call to Guid.ToString() will prevent the test from being run:

type SimpleRecord = { A: int; B: int } 
                    override x.ToString() = x.A.ToString() + Guid.NewGuid().ToString()

With the above change, the test still appears in Test Explorer, but it will not be run. Even right-clicking on it and selecting Run Selected Tests will not execute the test. No build errors are reported.

I also experimented with using DateTime.ToString() instead of Guid.ToString(), but that also refuses to run:

type SimpleRecord = { A: int; B: int } 
                    override x.ToString() = x.A.ToString() + DateTime.Now.ToString()

Why would calling Guid.ToString() or DateTime.ToString() within the ToString override on the type being tested result in the test not running?

Ifligus
  • 158
  • 3
  • 10
  • 1
    This looks like a test runner issue to me. I can reproduce it with the NUnit Test Adapter for Visual Studio, but if I run the tests with nunit-console.exe, it works fine. It also works fine with TestDriven.Net. – Mark Seemann Aug 14 '15 at 18:10
  • I guess you're talking about the ReSharper test runner? – Matthias Aug 14 '15 at 22:02
  • @MarkSeemann You are correct! I've posted the explanation and solution below. – Ifligus Aug 15 '15 at 17:12
  • @Matthias I'm using Visual Studio 2013 Test Explorer with NUnit Test Adapter. – Ifligus Aug 15 '15 at 17:16

1 Answers1

2

Charlie Poole provided an explanation of the root cause:

When tests are run using the NUnit console or gui runners, they are first loaded (discovered) and then executed. When running under the adapter, the tests are loaded twice, once in the discovery phase and again at the beginning of the execution phase. This is a result of the way the VS Test Window works: discovery and execution are run in separate processes created by VS.

This doesn't cause a problem in most cases. The tests are loaded twice, which means the code for creating any test data is run twice. Usually, the same data is generated both times.

However, when using random data, it's known that different data will be generated in each phase. Tests that are shown in the initial discovery seem to disappear while "unknown" tests are run.

In your case, the guid is generated twice. The first one is used as part of the name of the test, consequently, that test disappears in the execution phase.

Here's an example that demonstrates the workaround:

type ProblemRecord = { A: int; B: int } 
                     override x.ToString() = Guid.NewGuid().ToString()

[<TestFixture>]
type ``Test Cases``() =     
    // Using TestCaseData and explicitly setting a name will resolve the problem
    member x.SolutionDataSource =
        [
            TestCaseData(({ A = 1; B = 2} : ProblemRecord), 3)
                .SetName("Workaround")
        ]

    // This test case will be run by Test Explorer
    [<TestCaseSource("SolutionDataSource")>]
    member x.``SolutionTest`` (data: ProblemRecord, expected: int) =
        data.A + data.B
        |> should equal expected
Ifligus
  • 158
  • 3
  • 10