MemberData tests show up as one test instead of many

Question

When you use [Theory] together with [InlineData] it will create a test for each item of inline data that is provided. However, if you use [MemberData] it will just show up as one test.

Is there a way to make [MemberData] tests show up as multiple tests?

Answer 1

I spent a lot of time trying to figure this one out in my project. This related Github discussion from @NPadrutt himself helped a lot, but it was still confusing.

The tl;dr is this: [MemberInfo] will report a single group test unless the provided objects for each test can be completely serialized and deserialized by implementing IXunitSerializable.

---

Background

My own test setup was something like:

public static IEnumerable<object[]> GetClients()
{
    yield return new object[] { new Impl.Client("clientType1") };
    yield return new object[] { new Impl.Client("clientType2") };
}

[Theory]
[MemberData(nameof(GetClients))]
public void ClientTheory(Impl.Client testClient)
{
    // ... test here
}

The test ran twice, once for each object from [MemberData], as expected. As @NPadrutt experienced, only one item showed up in the Test Explorer, instead of two. This is because the provided object Impl.Client was not serializable by either interface xUnit supports (more on this later).

In my case, I didn't want to bleed test concerns into my main code. I thought I could write a thin proxy around my real class that would fool the xUnit runner into thinking it could serialize it, but after fighting with it for longer than I'd care to admit, I realized the part I wasn't understanding was:

The objects aren't just serialized during discovery to count permutations; each object is also deserialized at test run time as the test starts.

So any object you provide with [MemberData] must support a full round-trip (de-)serialization. This seems obvious to me now, but I couldn't find any documentation on it while I was trying to figure it out.

---

Solution

• Make sure every object (and any non-primitive it may contain) can be fully serialized and deserialized. Implementing xUnit's IXunitSerializable tells xUnit that it's a serializable object.

• If, as in my case, you don't want to add attributes to the main code, one solution is to make a thin serializable builder class for testing that can represent everything needed to recreate the actual class. Here's the above code, after I got it to work:

TestClientBuilder

public class TestClientBuilder : IXunitSerializable
{
    private string type;

    // required for deserializer
    public TestClientBuilder()
    {
    }

    public TestClientBuilder(string type)
    {
        this.type = type;
    }

    public Impl.Client Build()
    {
        return new Impl.Client(type);
    }

    public void Deserialize(IXunitSerializationInfo info)
    {
        type = info.GetValue<string>("type");
    }

    public void Serialize(IXunitSerializationInfo info)
    {
        info.AddValue("type", type, typeof(string));
    }

    public override string ToString()
    {
        return $"Type = {type}";
    }
}

Test

public static IEnumerable<object[]> GetClients()
{
    yield return new object[] { new TestClientBuilder("clientType1") };
    yield return new object[] { new TestClientBuilder("clientType2") };
}

[Theory]
[MemberData(nameof(GetClients))]
private void ClientTheory(TestClientBuilder clientBuilder)
{
    var client = clientBuilder.Build();
    // ... test here
}

It's mildly annoying that I don't get the target object injected anymore, but it's just one extra line of code to invoke my builder. And, my tests pass (and show up twice!), so I'm not complaining.

Answer 2

MemberData can work with properties or methods which return IEnumerable of object[]. You will see a separate test result for each yield in this scenario:

public class Tests
{ 
    [Theory]
    [MemberData("TestCases", MemberType = typeof(TestDataProvider))]
    public void IsLargerTest(string testName, int a, int b)
    {
        Assert.True(b>a);
    }
}

public class TestDataProvider
{
    public static IEnumerable<object[]> TestCases()
    {
        yield return new object[] {"case1", 1, 2};
        yield return new object[] {"case2", 2, 3};
        yield return new object[] {"case3", 3, 4};
    }
}

However, as soon as you will need to pass complex custom objects no matter how many test cases you will have the test output window will show just one test. This is not ideal behaviour and indeed very inconvenient while debugging which test case is failing. The workaround is to create your own wrapper which will derive from IXunitSerializable.

public class MemberDataSerializer<T> : IXunitSerializable
    {
        public T Object { get; private set; }

        public MemberDataSerializer()
        {
        }

        public MemberDataSerializer(T objectToSerialize)
        {
            Object = objectToSerialize;
        }

        public void Deserialize(IXunitSerializationInfo info)
        {
            Object = JsonConvert.DeserializeObject<T>(info.GetValue<string>("objValue"));
        }

        public void Serialize(IXunitSerializationInfo info)
        {
            var json = JsonConvert.SerializeObject(Object);
            info.AddValue("objValue", json);
        }
    }

Now you can have your custom objects as parameters to Xunit Theories and still see/debug them as independent results in the test runner window:

public class UnitTest1
{
    [Theory]
    [MemberData("TestData", MemberType = typeof(TestDataProvider))]
    public void Test1(string testName, MemberDataSerializer<TestData> testCase)
    {
        Assert.Equal(1, testCase.Object.IntProp);
    }
}

public class TestDataProvider
{
    public static IEnumerable<object[]> TestData()
    {
        yield return new object[] { "test1", new MemberDataSerializer<TestData>(new TestData { IntProp = 1, StringProp = "hello" }) };
        yield return new object[] { "test2", new MemberDataSerializer<TestData>(new TestData { IntProp = 2, StringProp = "Myro" }) };      
    }
}

public class TestData
{
    public int IntProp { get; set; }
    public string StringProp { get; set; }
}

Hope this helps.

Answer 3

In my recent project I experienced the very same problem and after some research the solution which I came up with is as follows:

Implement your custom MyTheoryAttribute extending FactAttribute along with MyTheoryDiscoverer implementing IXunitTestCaseDiscoverer and several custom MyTestCases extending TestMethodTestCase and implementing IXunitTestCase to your liking. Your custom test cases should be recognized by MyTheoryDiscoverer and used to encapsulate your enumerated theory test cases in form visible to Xunit framework even if values passed are not serialized natively by Xunit and do not implement IXunitSerializable.

What is most important there is no need to change your precious code under test!

It's a bit of work to do but since it was done already by me and is available under MIT license feel free to use it. It is part of DjvuNet project which is hosted on GitHub.

Direct link to the relevant folder with Xunit support code is below:

DjvuNet test support code

To use it either create separate assembly with this files or include them directly into your test project.

Usage is exactly the same as with Xunit TheoryAttribute and both ClassDataAttribute and MemberDataAttribute are supported i.e.:

[DjvuTheory]
[ClassData(typeof(DjvuJsonDataSource))]
public void InfoChunk_Theory(DjvuJsonDocument doc, int index)
{
    // Test code goes here
}


[DjvuTheory]
[MemberData(nameof(BG44TestData))]
public void ProgressiveDecodeBackground_Theory(BG44DataJson data, long length)
{
    // Test code goes here
}

Credit goes as well to another developer but unfortunately I cannot find his repo on github

Answer 4

For now, ReSharper can show all MemberData tests with custom parameters when your custom classes overrides ToString().

For example :

public static TheoryData<Permission, Permission, Permission> GetAddRuleData()
{
    var data = new TheoryData<Permission, Permission, Permission>
    {
        {
            new Permission("book", new[] {"read"}, null),
            new Permission("book", new[] {"delete"}, new[] {"2333"}),
            new Permission("book", new[] {"delete", "read"}, new[] {"*", "2333"})
        },
        {
            new Permission("book", new[] {"read"}, null),
            new Permission("music", new[] {"read"}, new[] {"2333"}), new Permission
            {
                Resources = new Dictionary<string, ResourceRule>
                {
                    ["book"] = new ResourceRule("book", new[] {"read"}, null),
                    ["music"] = new ResourceRule("music", new[] {"read"}, new[] {"2333"}),
                }
            }
        }
    };
    return data;
}

Permission overrides ToString(), then in ReSharper Test Session Explorer:

Answer 5

A simple alternative is that if you are using .net core project, instead of using vstest explorer you can run your tests in command line using "dotnet test"

The results are:

• get the total amount of tests
• amount that passed
• amount that failed

For member data tests that fail, you will get the associated parameter values for each of the failed member data test