Skip Fact/Theory when precondition is not satisfied

Question

Using xUnit 2.4.1, I'm looking for a way to make it do my twisted bidding when it comes to dynamically skipping integration tests.

I'm aware xUnit is not designed for integration testing, but I have no intention to use xUnit for some tests and Not-xUnit for other tests. I think the distinction between unit and integration tests is pretty useless for a test runner.

With that out of the way, I have some tests that depend on a database. I added traits to the tests as follows.

public class Some_database_interaction
{
    [Fact]
    [Trait("Demand", "SqlServer")]
    public async Task One_thing_is_inserted()
    {
        using var connection = await Connect();
        var applicationService = new SomeService(connection);
        
        var resultCount = await applicationService.DoTheImportantThing("some args", true);

        Assert.Equal(1, resultCount);
    }
}

Currently I'm using a command to skip tests that depend on the database when I don't want them to run.

dotnet test --filter Demand!=SqlServer

This is fine for use in a CI pipeline, but I cannot expect team members to type this command every time.

Instead I want some sort of way to detect capabilities and then skip tests if a demand is not satisfied. I don't know how to do that, though.

So far my best attempt is to add noise to my tests in the form of catching the error and short-circuiting the test.

SqlConnection connection;
try
{
    connection = await Connect();
}
catch
{
    // Exit the test
    return;
}

using connection;
var applicationService = new SomeService(connection);

But this has the undesired side-effect of showing test as successful instead of skipped.

Any hints on where to go from here? I know xUnit 3 will have Assert.Skip() but for now I am stuck on xUnit 2.

Can you group the tests to skip in a particular class? Then you can throw an exception in its constructor when the database isn't there and none of its tests will run. — Christopher Hamkins, Jul 25 '22 at 15:41
Another possible alternative: Change the tests from ``[Fact]`` to ``[Theory]`` with the connection as a parameter to the test method. Use a ``[ClassData(ConnectionProvider)]`` attribute to provide the connection to them. Implement the ``ConnectionProvider`` such that it returns an empty enumeration when there is no database and otherwise the single connection you constructed here in the method. (Caveat: I've never done this with generic objects as parameters, only with strings, so there may be some snags getting it to work...). Then there will simply be no tests when there is no database. — Christopher Hamkins, Jul 25 '22 at 15:49
Thanks for the suggestions. I tried throwing in the constructor but that also has the same effect (failed test instead of skipped—`Total tests: 1. Passed: 0. Failed: 1. Skipped: 0`). Using a Theory might be viable, though it also doesn't mark those tests as skipped. — Steven Liekens, Jul 26 '22 at 08:06
Does this help: https://stackoverflow.com/questions/4421328/how-do-i-skip-specific-tests-in-xunit-based-on-current-platform — Christopher Hamkins, Jul 26 '22 at 11:20

Christopher Hamkins · Accepted Answer · 2022-07-26T11:45:31.797

Here is another try based on How do I skip specific tests in xUnit based on current platform that I think does what you want.

It involves deriving a class from FactAttribute which can set its Skip attribute based on whether a connection is available or not.

using System;
using Xunit;

namespace SO73109781_skipping_unit_tests
{
    public class ConnectionUnitTests
    {
        [FactSkippedOnNoConnection]
        public void MyUnitTest()
        {
            Assert.Equal(1, 1);
        }
    }

    public class FactSkippedOnNoConnectionAttribute : FactAttribute
    {
        public FactSkippedOnNoConnectionAttribute()
        {
            try
            {
                // Simulate no connection by throwing here
                throw new Exception();
                // connection = await Connect();
            }
            catch
            {
                Skip = "No connection";
            }
        }
    }
}

When the method lands in the catch it sets the Skip attribute and the test is skipped and shown as such in the test explorer. When the try is successful, the test is run.

This is what I ended up implementing because of time constraints, but I'm missing the ability to access Traits. I was hoping to get the `"Demand"` trait of the current test inside my custom FactAttribute but it 's not possible. I also had to do a bunch of ugly things with `static Lazy` to avoid testing the connection over and over for each test. — Steven Liekens, Jul 28 '22 at 13:09
If I had more time, I would like to do a deep dive and implement a custom XunitTestCaseDiscoverer like in this sample https://github.com/xunit/samples.xunit/tree/main/DynamicSkipExample — Steven Liekens, Jul 28 '22 at 13:20

Christopher Hamkins · Answer 2 · 2022-07-26T11:34:47.843

To see what would happen, I mocked up my second suggestion as follows:

using System;
using System.Collections;
using System.Collections.Generic;
using Xunit;

namespace SO73109781_skipping_unit_tests
{
    public class ConnectionUnitTests
    {
        [Theory]
        [ClassData(typeof(ConnectionChecker))]
        public void MyUnitTest(string connected)
        {
            Assert.Equal(1, 1);
        }
    }


    public class ConnectionChecker : IEnumerable<object[]>
    {
        private static bool? connectionAvailable = null;

        public ConnectionChecker()
        {

            if (connectionAvailable == null)
            {
                /* What you would want to do here:
                SqlConnection connection;
                try
                {
                    connection = await Connect();
                    connectionAvailable = true;
                }
                catch
                {
                    connectionAvailable = false;
                }
                */ 
            }
            // For demonstration just set to specific value
            connectionAvailable = false;
        }

        public IEnumerator<object[]> GetEnumerator()
        {
            if (connectionAvailable ?? false)
            {
                yield return new string[] { "Connected" };
            }
        }

        IEnumerator IEnumerable.GetEnumerator() => GetEnumerator();

    }

}

When connectionAvailable = true; is used, the unit tests show up in the explorer and can be run:

When it is false the test shows up without any data:

When you run the test it fails with System.InvalidOperationException : No data found for SO73109781_skipping_unit_tests.ConnectionUnitTests.MyUnitTest, so I suppose this doesn't satisfy your requirement that the test not show up as failed.

One final suggestion: if you're willing to bring down the whole test process, you can throw an unhandled exception in the constructor of your test class when there is no connection available. None of the tests will run, they will be marked as "not run", but any tests from other projects that had not yet run will also not run. (I discovered this by accident when my application was throwing an unhandled exception.)

Skip Fact/Theory when precondition is not satisfied

2 Answers2