1

I have a collection of MsTest and NUnit tests that are being run in TFS 2015 using the VS Test step. I'm using NUnit Test Adapter 3.4.1 to run the tests on the build agent.

At the end, even though the tests pass, Nunit seems to croak and the build step fails with these two errors.

2016-09-04T09:59:44.7209787Z ##[error]Error: Exception NUnit.Engine.NUnitEngineException, Exception thrown executing tests
2016-09-04T09:59:44.7209787Z ##[error]
2016-09-04T09:59:44.7209787Z ##[error]Error: Exception encountered unloading AppDomain
2016-09-04T09:59:44.7209787Z ##[error]
2016-09-04T09:59:44.7209787Z Information: NUnit Adapter 3.4.1.0: Test execution complete
2016-09-04T09:59:44.8615975Z Results File: C:\agent\_work2\1\TestResults\SRV-BLD1 2016-09-04 01_22_45.trx
2016-09-04T09:59:44.8615975Z Total tests: 139. Passed: 134. Failed: 0. Skipped: 5.

I've checked that there are indeed 139 tests in the suite, and 5 are set to ignore (2 are MSTest, and 3 are NUnit).

I'm not sure if there's a place to get more detailed explanation of the error. Searching this site and google seems to suggest that the NUnit.Engine.NUnitEngineException is linked to test discovery (here, and here for instance), but my tests are being discovered, so I'm not sure if this is related (Pretty new to NUnit, so not sure of a lot of things).

I also saw two links dealing with Adapter failures (here and here), but the errors don't quite match up, although maybe just because I don't have the same level of verbosity.

In TFS, the step doesn't have any configuration on it, just that it's enabled and the path to the DLLs.

Does anyone know what is causing the errors (which I'm assuming are causing the build to fail)? Alternatively, what should be the next steps in getting a more precise/verbose error stack to investigate the issue?

As a side note, I saw this SO answer, which states this:

MSTest.exe returned an exit code of 1 indicating that not all tests passed.

I wasn't able to find any confirmation that VSTests fails when it encounters Skipped tests, but could this also be an issue?

Thank you for any help.

UPDATE

As suggested below, I tried running this from the IDE directly, and got this output (folders redacted)

------ Discover test started ------
NUnit Adapter 3.4.1.0: Test discovery starting

NUnit Adapter 3.4.1.0: Test discovery complete
========== Discover test finished: 139 found (0:00:00.8820879) ==========
------ Run test started ------
System.AppDomainUnloadedException: Attempted to access an unloaded AppDomain. This can happen if the test(s) started a thread but did not stop it. Make sure that all the threads started by the test(s) are stopped before completion.
NUnit Adapter 3.4.1.0: Test execution started
Running all tests in C:\agent\_work2\1\s\codePorject\bin\Debug\codeProjectTests.dll
NUnit3TestExecutor converted 37 of 37 NUnit test cases
System.AppDomainUnloadedException: Attempted to access an unloaded AppDomain. This can happen if the test(s) started a thread but did not stop it. Make sure that all the threads started by the test(s) are stopped before completion.
Exception NUnit.Engine.NUnitEngineException, Exception thrown executing tests
Exception encountered unloading AppDomain
NUnit Adapter 3.4.1.0: Test execution complete
========== Run test finished: 139 run (1:20:10.3290294) ==========

I've found a similar xUnit issue open, but it doesn't seem to have a solution.

This StackOverflow answer suggests to use a sleep timer, so I might try that.

Community
  • 1
  • 1
Kolichikov
  • 2,944
  • 31
  • 46

2 Answers2

1

In the end, it looks like some sort of a race condition between Firefox browser windows and nUnit. In my cleanup code, I'm killing firefox and iisexpress processes. Adding a sleep call has eliminated the issue:

public static void AssemblyCleanup()
{
    foreach (var process in Process.GetProcessesByName("firefox")) process.Kill();
    foreach (var process in Process.GetProcessesByName("iisexpress")) process.Kill();
    System.Threading.Thread.Sleep(5000);
}
shytikov
  • 9,155
  • 8
  • 56
  • 103
Kolichikov
  • 2,944
  • 31
  • 46
0

An NUnitEngineException is what it says: an exception that was discovered by the engine. It can be caused by many things and the message indicates the problem. In your case, the message says "Exception encountered unloading AppDomain" which means... well what it says.

The test adapter catches and handles the exception, producing the message you see. There is some indication that TFS also sees the exception and fails test run as a result. If you were to run this under the VS IDE, I think you would see the message from the adapter but the run would not fail. There is an NUnit3 Test Adapter issue about this, but it's not clear if the solution lies within the adapter or if it's a TFS problem.

If you do try this under the IDE I hope you will post the output window text from that run as well.

Charlie
  • 12,928
  • 1
  • 27
  • 31
  • So should I be looking in the Output - Tests when I run it in the IDE? I get the N3ExecutionAgent, it runs the tests, but there is no message about any errors. – Kolichikov Sep 05 '16 at 02:33
  • If you didn't see a message about a failure to unload the AppDomain, then it may only happen intermittently. The adapter definitely displays the message without regard to where it's running. Make sure you are using the same adapter version in each case. – Charlie Sep 05 '16 at 23:56