NUnit Test Debugging

Question

I'm working on a Geometry library. There are 200+ unit tests.

There's a particularly stubborn test that fails whenever I select "Run All", but the test passes when I run that test individually, or use the debugger on it. I do believe the issue showed up about when I shifted over from visual studio '13 to the '15 edition.

Now some notes about the geometry library:

The objects are immutable.

The tests have no shared objects between them.

So my Question: What are the possible causes for this odd behavior?

Edit:

[Test()]
public void Plane_IntersectionWithPlane_IdenticalPlane()
{
     Plane testPlane = new Plane(new Direction(Point.MakePointWithInches(2, -1, 1)), 
                                               Point.MakePointWithInches(2, 1, 2));
     Line found = (testPlane.Intersection(testPlane));

     Line expected = new Line(new Direction(Point.MakePointWithInches(0, -1, -1)), 
                                            Point.MakePointWithInches(2, 1, 2));
     Assert.IsTrue(found.Equals(expected));
}

Are there any well known quirks to using NUnit that might cause a test to fail when run as part of a group, vs. run by itself? — Charles Taylor, Jul 28 '15 at 17:32
None that I know of, and it's more likely you have some crosstalk or randomness in your tests. Can you post test/implementation code? — Patrick Quirk, Jul 28 '15 at 17:34
No, this is very likely on your side. Show a short, reproducable example that demonstrates the problem and we can look at the exact issue. Likely there's a shared resource, incorrect asynchronicity or a myriad of other possibilities. — Jeroen Vannevel, Jul 28 '15 at 17:41
Each test is designed to run independant of each other. There are no setup methods or shared objects between tests. — Charles Taylor, Jul 28 '15 at 17:56
I've seen this kind of behavior before, but I could usually not explain it. Which test runner are you using? Have you tried a different one? Does the code you're testing mutate some shared state (e.g. static fields) ? — Thomas Levesque, Aug 02 '15 at 21:03
If there's another test interacting with this one, you could perhaps track it down by running this test and 100 of the other tests. If it fails with one half but passes with the other then you can keep bisecting them until you find the culprit (of course there may be more than one). — Matthew Strawbridge, Aug 02 '15 at 22:11
I'm using NUnit. @Matthew: If I highlight any number of tests and choose "run selected tests", they all pass or fail as they should. I get the weird behavior from hitting the run all button, or equivalently the hot key (ctrl + R, A). — Charles Taylor, Aug 04 '15 at 17:42
Is there floating-point math involved? Are you comparing floats for *exact* equality? — Patrick Quirk, Aug 05 '15 at 15:46
All Equality operators (==, !=, <, > etc.) are overloaded to allow some leniency. How does this relate to the issue though? — Charles Taylor, Aug 05 '15 at 20:15
You are using Point class statically to call the function MakePointWithInches. Is the only case where one test can depend from another test that also use this funcition... Do you check it? Can you share the implementation of the Point class? — Ruben Aguilar, Aug 29 '15 at 22:49
One other question - are there any async methods anywhere within the function stack you're calling? One quick way to see if this is due to a shared function is to add a Thread.Sleep() statement at the beginning of the test. If it then passes, you know it's a problem with other tests being able to modify the state you have. — Mathieson, Aug 31 '15 at 18:42
Ain't this a logical problem? Intersecting a plane with itself does not result in a single line, it results in a plane which is an unlimited number of lines. Why do you think it should just be the one line you expect? — Thomas Weller, Sep 02 '15 at 19:39
IMHO you need to provide more details. With the given information there are too many possibilities: Threading, static members, caching, ... — Thomas Weller, Sep 02 '15 at 19:43
@Ruben: MakePointWithInches is literally a one line constructor call. Yes its a static method, but there are no static classes involved. — Charles Taylor, Sep 03 '15 at 00:44
You have to check not only the unit tests code (where the problem obviously is not found), but all code that is executed by the test. Check whether there is any static field somewhere, for instance for caching, performance optimizations or whatever. Is there anything in a initialization method? Do you use any kind of mocks? — Stefan Steinegger, Sep 04 '15 at 08:26
@Charles Taylor: Side note.. why is the test assuming a unique line as an intersection of a plane with itself? There would be an infinite number of lines to describe the intersection of a plane with itself. — Frank Bryce, Sep 04 '15 at 14:57

score 7 · Answer 1 · answered Sep 03 '15 at 14:06

I've come across similar issues in the past, but it's always turned out to be some unexpected interaction between the underlying code elements, or the way that the tests have been written. Some issues to check are:

Static object construction / destruction
Synchronisation issues
Mock setup / cleanup issues

I've found the best approach to track down the issue is follow an approach similar to that suggested by @Matthew Strawbridge in the comments. I add Ignore attributes to Tests/TestFixtures to remove tests until the run-all starts working and then start adding them back in until it breaks again narrowing down the problem.

Sometimes you will also find that ignoring the test that is currently failing will cause another test to fail instead. This is a good indication that the problem is actually being caused by another one of your tests not cleaning up after itself correctly.

Looking at the code between the tests that fail/seem to cause the fail can then help you to narrow down the interaction. The error/failure reason for the tests can also help of course...

Selecting 'Run-all' runs all of the tests in a predictable order, so the tests will generally be run the same way everytime. If you select a batch of tests then it's possible that the runner may choose to run them in a different order, based around your selection order which may be why you've experience different behaviour when selecting tests, rather than using run-all.

score 3 · Accepted Answer · answered Sep 01 '15 at 14:02

Try the following:

Open Nunit GUI And before you run the test change the above in the settings:

It helped me once detect my problems.

BTW: what version of NUNIT are you using ? If you don't know what is Nunit GUI then you probably didn't downloaded Nunit separately. You can get the install from here :

http://www.nunit.org/index.php?p=download

score 0 · Answer 3 · answered Aug 29 '15 at 22:07

From what I see in the testcase, a diverging behavior is strange, but the test failing is a perfectly valid result:

A plane intersected with itself would be the plane, so if the result is restricted to being a line, then any line with 2 distinct points on the plane would be a valid result, but testing for one particular line looks like reverse engineering of execution results into test expectations.

It would be a different story if the requirement / specification is very specific about the direction of the resulting lines from identity intersection, so maybe you want to provide some more insight in this area.

I'm sorry for posting this as answer rather than a comment - missing rep.

This is just an example there's other tests that have this odd behavior. In the case where a plane intersects self, it just picks a particular line in that plane. Whether or not that's a good way to handle the degeneracy is open to debate, but the line tested against is the line the method should have chosen. (and is the one it chooses when the test runs by itself) — Charles Taylor, Sep 01 '15 at 14:57

score 0 · Answer 4 · edited May 23 '17 at 12:32

Please check if there are any static variables involved. Static Variables are present at AppDomain level, therefor its a possibility that a static variable set by one test case may cause side effects to other test cases.

You may run this testcase in a separate app domain first to confirm the behaviour before starting the search for static variable. I'm sorry I never tried to create a new appdomain in nunit. But one of the answers here does give a hint about how to create a new appdomain for a test case - Run unit tests in different appdomain with NUnit

score 0 · Answer 5 · answered Sep 04 '15 at 15:05

If this test passes in isolation but fails when running alongside other tests then it is extremely likely that you have some shared state in your classes.

If you say that all our objects are immutable then my first guess would be to look at the implementation of

Line Intersection (Plane plane);

You expect this method to return a Line which is considered the same as your "expected" line, so probably Intersection returns a line based on some shared state.

Could you show us the implementation of Intersection?

NUnit Test Debugging

5 Answers5