Unit test fails when running all, but passes when running individually

Question

I have about 12 unit tests for different scenarios, and I need to call one async method in these tests (sometimes multiple times in one test). When I do "Run all", 3 of them will always fail. If I run them one by one using "Run selected test", they will pass. The exception in output I'm getting is this:

System.AppDomainUnloadedException: Attempted to access an unloaded AppDomain. This can happen if the test(s) started a thread but did not stop it. Make sure that all the threads started by the test(s) are stopped before completion.

I can't really share the code, as it's quite big and I don't know where to start, so here is example:

[TestMethod]
public async Task SampleTest()
{
    var someProvider = new SomeProvider();

    var result = await someProvider.IsSomethingValid();

    Assert.IsTrue(result == SomeProvider.Status.Valid);

    NetworkController.Disable();

    result = await someProvider.IsSomethingValid();

    Assert.IsTrue(result == SomeProvider.Status.Valid);

    NetworkController.Enable();
}

EDIT: The other 2 failing methods set time to the future and to the past respectively.

[TestMethod]
public async Task SetTimeToFutureTest()
{
    var someProvider = new SomeProvider();

    var today = TimeProvider.UtcNow().Date;

    var result = await someProvider.IsSomethingValid();

    Assert.IsTrue(result == SomeProvider.Status.Valid);

    TimeProvider.SetDateTime(today.AddYears(1));

    var result2 = await someProvider.IsSomethingValid();

    Assert.IsTrue(result2 == SomeProvider.Status.Expired);
}

Where TimeProvider looks like this:

public static class TimeProvider
{
    /// <summary> Normally this is a pass-through to DateTime.Now, but it can be overridden with SetDateTime( .. ) for testing or debugging.
    /// </summary>
    public static Func<DateTime> UtcNow = () => DateTime.UtcNow;

    /// <summary> Set time to return when SystemTime.UtcNow() is called.
    /// </summary>
    public static void SetDateTime(DateTime newDateTime)
    {
        UtcNow = () => newDateTime;
    }

    public static void ResetDateTime()
    {
        UtcNow = () => DateTime.UtcNow;
    }
}

EDIT 2:

    [TestCleanup]
    public void TestCleanup()
    {
        TimeProvider.ResetDateTime();
    }

Other methods are similar, I will simulate time/date change, etc.

I tried calling the method synchronously by getting .Result() out of it, etc, but it didn't help. I read ton material on the web about this but still struggling.

Did anyone run into the same problem? Any tips will be highly appreciated.

Answer 1

I can't see what you're doing with your test initialization or cleanup but it could be that since all of your test methods are attempting to run asynchronously, the test runner is not allowing all tasks to finish before performing cleanup.

Are the same few methods failing when you run all of the tests or is it random? Are you sure you are doing unit testing and not integration testing? The class "NetworkController" gives me the impression that you may be doing more of an integration test. If that were the case and you are using a common class, provider, service, or storage medium (database, file system) then interactions or state changes caused by one method could affect another test method's efficacy.

Answer 2

When running tests in async/await mode, you will incur some lag. It looks like all your processing is happening in memory. They're probably passing one an one-by-one basis because the lag time is minimal. When running multiple in async mode, the lag time is sufficient to cause differentiation in the time results.

I've run into this before doing NUnit tests run by NCrunch where a DateTime component is being tested. You can mitigate this by reducing the scope of your validation / expiration logic to match to second instead of millisecond, as long as this is permissible within your acceptance criteria. I can't tell from your code what the logic is driving validation status or expiration date, but I'm willing to bet the async lag is the root cause of the test failure when run concurrently.

Answer 3

Both tests shown use the same static TimeProvider, thus interference by methods like ResetDateTime in the cleanup and TimeProvider.SetDateTime(today.AddYears(1)); in a test are to be expected. Also the NetworkController seems to be a static resource, and connecting/disconnecting it could interfere with your tests.

You can solve the issues in several ways:

• get rid of static resources, use instances instead
• lock the tests such that only one test can be run at a time

Aside from that, almost every test framework offers more than just Assert.IsTrue. Doesn't your framework offer an Assert.AreEqual? That improves readabilty. Also, with more than one Assert in a test, custom messages indicating which of the test failed (or that an Assert is for pre-condition, not the actual test) are recommended.