Dynamic assertions based on [InlineData] or [MemberData]

Question

currently we are developing some bigger component tests. These are relying on some rather more complex data generators, which are creating data basis of some templates.

Problem

While testing with xUnit we feeding the test with the help of [InlineData]:

[Theory]
[InlineData(Device.Type1, DeviceConfig.VeryHotDevice)]
[InlineData(Device.Type1, DeviceConfig.VeryColdDevice)]
public async Task TestCalculation(string deviceType, DeviceConfig deviceConfig)

The problem I have is regarding the Assertions for the different DeviceConfig values. I have to create an switch-statement with cases for every single one in the test method:

switch (simConfig)
{
    case DeviceConfig.VeryHotDevice:
        AssertVeryHotDevice(resultData);
    break;
    case DeviceConfig.VeryColdDevice:
        AssertVeryVoldDevice(resultData);
    break;
}

What I would like to have is something like the following. So basically just pass the method with the assertions for the specific [InlineData] and .Invoke it inside the test method.

[Theory]
[InlineData(Device.Type1, DeviceConfig.VeryHotDevice, )]
[InlineData(Device.Type1, DeviceConfig.VeryColdDevice)]
public async Task TestCalculation(string deviceType, DeviceConfig deviceConfig, Action<ResultData> assertAction)

Without going into details I already got this working by changing [InlineData] to [MemberData] with help of this answer (Lambda expression as inline data in xUnit).

Sadly you then run into following issue (MemberData tests show up as one test instead of many), because the Actions are not IXunitSerializable.

For me it's neither acceptable to show only one test (while it in reality are three to four), nor to implement some unconvenient clutter code to support IXunitSerializable.

Question

How would you keep the assertions here some kind of dynamic and pass them directly with the test data with the goal to keep the test method clean? Of course I would also be happy, if you propose something completely different.

If there are 3 or 4 tests and they each use a different assertion method, is there any reason you don't just have 3 or 4 separate tests? Otherwise, I think you seem have included all your options, I'm not sure there's anything to add. — Charles Mager, Feb 20 '19 at 14:54
Yes I would consider separate test methods per config. Because it seems each config got its own assertion logic which might be correct or incorrect independently from others. So semantically seems to me these are different tests rather than one varied test. — Mohammad, Feb 20 '19 at 15:06

score 1 · Answer 1 · answered Oct 31 '19 at 08:39

If you want to have your tests show up, you will have to support IXunitSerializable I'm afraid. But that doesn't meet it has to become 'clutter'. We are using a fairly simple approach based on ClassData and IXunitSerializable.

The snippet below demonstrates how we write our tests. In my opinion it's clear and consice. We decorate our method with a ClassData attribute using a type that inherits from ClassDataCollection<T>. This class must implement the GetData() method where we return instances of T. This is necessary because in order to support ClassData we have to implement IEnumerable<object[]>. The base class is responsible for creating an object[1] for each item returned by GetData().

To have the tests show up individually, we need to implement IXunitSerializable on the TestItem class. In the code sample below, the TestItem inherits from a ClassDataItem class. This class implements IXunitSerializable, and it just uses JSon to get or add values onto the IXunitSerializationInfo.

This is what the test looks like:

       [Theory]
       [ClassData(typeof(RunTestWithMemberDataCollection))]
       public async Task RunTestWithMemberData(RunTestWithMemberDataCollection.TestItem data)
       {
       }

       public class RunTestWithMemberDataCollection :
           ClassDataCollection<RunTestWithMemberDataCollection.TestItem>
       {
           public override IEnumerable<TestItem> GetData()
           {
               yield return new TestItem { Prop1 = "", Prop2 = "data 1" };
               yield return new TestItem { Prop1 = "", Prop2 = "data 2" };
           }

           public class TestItem : ClassDataItem
           {
               public string Prop1 { get; set; }

               public string Prop2 { get; set; }
           }
       }

And here is the supporting code that you only need once:

    public abstract class ClassDataCollection<T> : IEnumerable<object[]>
        where T : ClassDataItem
    {
        public IEnumerator<object[]> GetEnumerator()
        {
            return GetData()
                .Select(d => new object[] { (IXunitSerializable)d })
                .GetEnumerator();
        }

        IEnumerator IEnumerable.GetEnumerator()
        {
            return GetEnumerator();
        }

        public abstract IEnumerable<T> GetData();
    }

    public class ClassDataItem : IXunitSerializable
    {
        public void Deserialize(IXunitSerializationInfo info)
        {
            JsonConvert.PopulateObject(info.GetValue<string>("TestItem"), this);
        }

        public void Serialize(IXunitSerializationInfo info)
        {
            info.AddValue("TestItem", JsonConvert.SerializeObject(this));
        }
    }

Dynamic assertions based on [InlineData] or [MemberData]

1 Answers1