Unit Tests failing when I Run All Tests but pass when I Debug

Question

I'm using NUnit3 in Visual Studio 2017 and doing TDD. Something really strange is happening since I updated my code to make my latest test pass.

Now, 3 of my other tests are failing when I click Run All Tests, as below:

It is telling me that the actual and expected values in my Assert method are not equal.

However, when I put a breakpoint at the line where the Assert method is and start debugging, the stacktrace is showing that expected and actual are the same value and then the test passes, as below:

Am I doing something stupid or could there be a bug in VS2017 or NUnit or something?

This ever happen to anyone else?

[Edit: I should probably add that I have written each test as a separate class]

@Zenima Yeah, all three pass when run individually but fail when I click 'Run All Tests' which I don't understand because they are still unit tests, not integration tests. — Michael Hennigan, May 13 '17 at 22:03
@MichaelHennigan The failing tests share a resource that affects them all when tested together. Recheck the affected tests and their subjects. — Nkosi, May 13 '17 at 22:25
@Nkosi So could this happen because I use the same variable names in different tests? — Michael Hennigan, May 13 '17 at 22:29
@MichaelHennigan if that variable is global to the test class yes. but can't see if that is your case without seeing the offending code — Nkosi, May 13 '17 at 22:37
Never Mind, I tried giving the variables in each test class different names but this didn't work. — Michael Hennigan, May 13 '17 at 22:41
@MichaelHennigan You should also look into **static** fields or properties in the subjects. They tends to cause issues when not designed properly. — Nkosi, May 13 '17 at 22:45
@Nkosi Okay, thanks for the advice. I did introduce a static field in a production class when I did the code change that caused this problem so maybe that's it — Michael Hennigan, May 13 '17 at 22:50

score 15 · Accepted Answer · answered May 13 '17 at 22:52

15

The failing tests share a resource that affects them all when tested together. Recheck the affected tests and their subjects.

You should also look into static fields or properties in the subjects. They tends to cause issues if not used properly when designing your classes.

answered May 13 '17 at 22:52

Nkosi

235,767
35
427
472

Thanks @Nkosi. When I commented out the static field and hit Run All Tests, the two older tests passed. However, my latest test is failing. I was using a static int nextID to give new user objects a UserID of NextID and then incrementing NextID. How can I achieve this functionality without messing up my unit tests? – Michael Hennigan May 13 '17 at 22:57
@MichaelHennigan Ok that is a new issue. Ask a new question that properly explains the issue and I'll see how best I can help. I have some ideas but it would only be a guess without seeing the actual problem you are encountering. This is most probably a design issue. – Nkosi May 13 '17 at 22:59
Or does it even matter? Should I just ignore the fact that my tests don't all pass when I Run All? – Michael Hennigan May 13 '17 at 22:59
Do not ignore tests. designing your classes properly will fix the problem. Never ignore tests. they will come back to bite you (technical debt) in the future. – Nkosi May 13 '17 at 23:00
Okay - new question here: http://stackoverflow.com/questions/43958823/how-do-i-use-a-static-variable-without-messing-up-my-unit-tests – Michael Hennigan May 13 '17 at 23:10

score 1 · Answer 2 · answered May 13 '17 at 21:18

1

Some subtle differences might occur. For instance if a first test change a state which affects the behavior of a second test, then the outcome of this 2nd test may not be the same if I run it alone.

An idea to help understand a test failure when a breakpoint can't be used, could be to add logging.

Anyway, to answer your questions:

This ever happen to anyone else?

Yes

Am I doing something stupid or could there be a bug in VS2017 or NUnit or something?

I bet that it's neither: just a case a bit more subtle

answered May 13 '17 at 21:18

gturri

13,807
9
40
57

Thanks for getting back to my question. I thought there might be some odd but logical reason due to my code change but I have spent ages trawling through both my test code and production code and can't find a reason why the tests fail. What happens you click 'Run All Tests'? I don't understand why it is different than running each test one by one – Michael Hennigan May 13 '17 at 22:07
Think of it this way: One of your tests interferes with the others. They can only interfere if you run them all together. – Charlie May 14 '17 at 00:09

score 0 · Answer 3 · answered Oct 04 '18 at 15:42

I experienced a similar issue in Visual Studio 2017 using MSTest as the testing framework. Assertions in unit tests were failing when the tests were run but would pass when the unit tests were debugged. This was occurring for a handful of unit tests but not all of them. In addition to the Assertion failures, many of the units tests were also failing due to a System.TypeLoadException (Could not load type from assembly error). I ultimately did the following which solved the problem:

Open the Local.testsettings file in the solution
Go to the "Unit Test" settings
Uncheck the "Use the Load Context for assemblies in the test directory." checkbox

After taking these steps all unit tests started passing when run.

score 0 · Answer 4 · answered Dec 31 '19 at 16:56

I encountered this phenomenon myself, but found the cause quite easily. More concretely, I tested some matrix calculations, and in my test class I defined data to calculate with as a class variable and performed my calculations with it. My matrix routines, however, modified the original data, so when I used "run tests" on the test class, the first test corrupted the data, and the next test could not succeed.

The sample code below is an attempt to show what I mean.

[TestFixture]
public void MyTestClass()
{
    [Test]
    public void TestMethod1()
    {
        MyMatrix m = new MyMatrix();

        // Method1() modifies the data...
        m.Method1(_data);
    }

    [Test]
    public void TestMethod2()
    {
        MyMatrix m = new MyMatrix();

        // here you test with modified data and, in general, cannot expect success
        m.Method2(_data);
    }

    // the data to test with
    private double[] _data = new double[1, 2, 3, 4]{};
}

Unit Tests failing when I Run All Tests but pass when I Debug

4 Answers4

Linked