1

I have 3 NUnit test assemblies are testing my custom code written for the Kentico CMS platform. I can run these tests both in Visual Studio (using the NUnit adapter) and the NUnit console runner. I'm now trying to have those tests run on my TeamCity build server. I've got the NUnit runner setup and it is returning some results, but I'm finding that not all the tests are being executed. In one case, one of my assemblies reports having no test fixtures despite the fact I know there definitely are.

System Details

  • TeamCity v10.0.5 (build 42677)
  • Nunit 3.8.1
  • Nunit.ConsoleRunner 3.8.0
  • Kentico 11
    • Kentico.Libraries - 11.0.35
    • Kentico.Libraries.Tests - 11.0.35

Problem 1

I have a test assembly called gto.ecommerce.core.tests.dll. When this is run on the TeamCity server, I get this output (straight from the NUnit console runner):

NUnit Console Runner 3.8.0
Copyright (c) 2018 Charlie Poole, Rob Prouse

Runtime Environment
   OS Version: Microsoft Windows NT 6.3.9600.0
  CLR Version: 4.0.30319.42000

Test Files
    D:\TeamCity\buildAgent\work\4a231fb0e41e27f5\Tests\gto.ecommerce.core.tests\bin\Build\gto.ecommerce.core.tests.dll


Run Settings
    DisposeRunners: True
    WorkDirectory: D:\TeamCity\buildAgent\work\4a231fb0e41e27f5\packages\NUnit.ConsoleRunner.3.8.0\tools
    ImageRuntimeVersion: 4.0.30319
    ImageTargetFrameworkName: .NETFramework,Version=v4.6
    ImageRequiresX86: False
    ImageRequiresDefaultAppDomainAssemblyResolver: False
    NumberOfTestWorkers: 8

Test Run Summary
  Overall result: Passed
  Test Count: 6, Passed: 6, Failed: 0, Warnings: 0, Inconclusive: 0, Skipped: 0
  Start time: 2019-01-15 10:16:53Z
    End time: 2019-01-15 10:16:54Z
    Duration: 1.219 seconds

Results (nunit3) saved as TestResult.xml

These are exactly the same results returned when the TeamCity build step is run as well.

However, there should be more tests here. If I copy the binaries that TeamCity built to my local machine and run the same NUnit console runner command, I get these results:

NUnit Console Runner 3.8.0 
Copyright (c) 2018 Charlie Poole, Rob Prouse

Runtime Environment
   OS Version: Microsoft Windows NT 10.0.14393.0
  CLR Version: 4.0.30319.42000

Test Files
    C:\temp\gto-gtoengineering\TeamCity\gto.ecommerce.core.tests\Build\gto.ecommerce.core.tests.dll


Run Settings
    DisposeRunners: True
    WorkDirectory: Z:\
    ImageRuntimeVersion: 4.0.30319
    ImageTargetFrameworkName: .NETFramework,Version=v4.6
    ImageRequiresX86: False
    ImageRequiresDefaultAppDomainAssemblyResolver: False
    NumberOfTestWorkers: 8

Test Run Summary
  Overall result: Passed
  Test Count: 33, Passed: 33, Failed: 0, Warnings: 0, Inconclusive: 0, Skipped: 0
  Start time: 2019-01-15 10:19:31Z
    End time: 2019-01-15 10:19:35Z
    Duration: 4.494 seconds

Results (nunit3) saved as TestResult.xml

Notice how my machine says there are 33 tests overall, which is the correct number.

Problem 2

If I run the same scenarios on another assembly I have, rwy.common.core.tests, then this is the TeamCity server result:

NUnit Console Runner 3.8.0
Copyright (c) 2018 Charlie Poole, Rob Prouse

Runtime Environment
   OS Version: Microsoft Windows NT 6.3.9600.0
  CLR Version: 4.0.30319.42000

Test Files
    D:\TeamCity\buildAgent\work\4a231fb0e41e27f5\Tests\rwy.common.core.tests\bin\Build\rwy.common.core.tests.dll


Errors, Failures and Warnings

1) Invalid : D:\TeamCity\buildAgent\work\4a231fb0e41e27f5\Tests\rwy.common.core.tests\bin\Build\rwy.common.core.tests.dl
l
Has no TestFixtures

Run Settings
    DisposeRunners: True
    WorkDirectory: D:\TeamCity\buildAgent\work\4a231fb0e41e27f5\packages\NUnit.ConsoleRunner.3.8.0\tools
    ImageRuntimeVersion: 4.0.30319
    ImageTargetFrameworkName: .NETFramework,Version=v4.6
    ImageRequiresX86: False
    ImageRequiresDefaultAppDomainAssemblyResolver: False
    NumberOfTestWorkers: 8

Test Run Summary
  Overall result: Failed
  Test Count: 0, Passed: 0, Failed: 0, Warnings: 0, Inconclusive: 0, Skipped: 0
  Start time: 2019-01-15 10:21:16Z
    End time: 2019-01-15 10:21:17Z
    Duration: 1.046 seconds

Results (nunit3) saved as TestResult.xml

But again, I copy the same binaries to my local machine and I get this result:

NUnit Console Runner 3.8.0 
Copyright (c) 2018 Charlie Poole, Rob Prouse

Runtime Environment
   OS Version: Microsoft Windows NT 10.0.14393.0
  CLR Version: 4.0.30319.42000

Test Files
    C:\temp\gto-gtoengineering\TeamCity\rwy.common.core.tests\Build\rwy.common.core.tests.dll


Run Settings
    DisposeRunners: True
    WorkDirectory: Z:\
    ImageRuntimeVersion: 4.0.30319
    ImageTargetFrameworkName: .NETFramework,Version=v4.6
    ImageRequiresX86: False
    ImageRequiresDefaultAppDomainAssemblyResolver: False
    NumberOfTestWorkers: 8

Test Run Summary
  Overall result: Passed
  Test Count: 20, Passed: 20, Failed: 0, Warnings: 0, Inconclusive: 0, Skipped: 0
  Start time: 2019-01-15 10:22:55Z
    End time: 2019-01-15 10:23:00Z
    Duration: 4.242 seconds

Results (nunit3) saved as TestResult.xml

There should definitely be 20 tests available, but when run on the TeamCity server it says it cannot find any.

Summary

So does anyone know why two different machines running what I assume to be the same versions of NUnit and console runner produce vastly different results? I cannot see any obvious differences in version numbers apart from Windows, but the same .NET framework should be being used I think.

I do use a significant number of TestCaseSource and TestFixtureSource attributes to increase my test count for parameterized tests, but I'm not sure that is the cause - if it works on one machine I cannot see why another would be different.

Peter Monks
  • 4,219
  • 2
  • 22
  • 38
  • There's no reason this would happen, without good reason! I suggest you diff the two TestResult.xml files, and look at what's different between the two runs! – Chris Jan 15 '19 at 22:09
  • 1
    @Chris And it turns out there was good reason (albeit well hidden). Updated both my question and provided my solution below – Peter Monks Jan 16 '19 at 12:51
  • the error you described below should definitely show on the console output! Would you be able to file an issue on the nunit project, including your logs? Thanks! – Chris Jan 17 '19 at 20:48
  • 1
    @Chris I'm trying to make a simple project that replicates the issue, but now I'm finding I cannot reproduce the error. Perhaps the above is happening because I'm forced to use an older version of `NUnit.ConsoleRunner` - 3.8.0, because of the version of TeamCity I'm using. I'm going to assume that newer versions have fixed this issue already – Peter Monks Jan 23 '19 at 13:45

1 Answers1

1

After quite a bit of tracing I found the cause not to be NUnit or TeamCity, but rather it was the fact I was using libraries from Kentico, specifically I was using Kentico's CMS.Tests library for helping unit test my custom Kentico code as explained here.

Because of this I've updated my question to be more specific to Kentico, and will provide my solution for fixing this below.

Identify Root Cause

After a bit of internet searching I found that the NUnit console runner has a --trace command line option. By passing in --trace=Verbose I was able to get trace files written to the working directory set for the console runner and then compare the two machines. My local development machine showed that all fixtures were being found correctly, but the TeamCity server would produce trace files that had output similar to this:

InternalTrace: Initializing at level Debug
14:41:48.559 Debug [ 5] DefaultTestAssemblyBuilder: Loading D:\TeamCity\buildAgent\work\4a231fb0e41e27f5\Tests\rwy.common.core.tests\bin\Build\rwy.common.core.tests.dll in AppDomain domain-
14:41:48.570 Debug [ 5] DefaultTestAssemblyBuilder: Examining assembly for test fixtures
14:41:48.579 Debug [ 5] DefaultTestAssemblyBuilder: Found 12 classes to examine
14:41:48.691 Error [ 5] DefaultTestAssemblyBuilder: System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.VisualStudio.QualityTools.UnitTestFramework, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.
File name: 'Microsoft.VisualStudio.QualityTools.UnitTestFramework, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'
   at System.ModuleHandle.ResolveType(RuntimeModule module, Int32 typeToken, IntPtr* typeInstArgs, Int32 typeInstCount, IntPtr* methodInstArgs, Int32 methodInstCount, ObjectHandleOnStack type)
   at System.ModuleHandle.ResolveTypeHandleInternal(RuntimeModule module, Int32 typeToken, RuntimeTypeHandle[] typeInstantiationContext, RuntimeTypeHandle[] methodInstantiationContext)
   at System.Reflection.RuntimeModule.ResolveType(Int32 metadataToken, Type[] genericTypeArguments, Type[] genericMethodArguments)
   at System.Reflection.CustomAttribute.FilterCustomAttributeRecord(CustomAttributeRecord caRecord, MetadataImport scope, Assembly& lastAptcaOkAssembly, RuntimeModule decoratedModule, MetadataToken decoratedToken, RuntimeType attributeFilterType, Boolean mustBeInheritable, Object[] attributes, IList derivedAttributes, RuntimeType& attributeType, IRuntimeMethodInfo& ctor, Boolean& ctorHasParameters, Boolean& isVarArg)
   at System.Reflection.CustomAttribute.IsCustomAttributeDefined(RuntimeModule decoratedModule, Int32 decoratedMetadataToken, RuntimeType attributeFilterType, Int32 attributeCtorToken, Boolean mustBeInheritable)
   at System.Reflection.CustomAttribute.IsDefined(RuntimeMethodInfo method, RuntimeType caType, Boolean inherit)
   at NUnit.Framework.Internal.Reflect.GetMethodsWithAttribute(Type fixtureType, Type attributeType, Boolean inherit)
   at NUnit.Framework.Internal.TestFixture..ctor(ITypeInfo fixtureType, Object[] arguments)
   at NUnit.Framework.Internal.Builders.DefaultSuiteBuilder.BuildFrom(ITypeInfo typeInfo)
   at NUnit.Framework.Api.DefaultTestAssemblyBuilder.GetFixtures(Assembly assembly, IList names)

WRN: Assembly binding logging is turned OFF.
To enable assembly bind failure logging, set the registry value [HKLM\Software\Microsoft\Fusion!EnableLog] (DWORD) to 1.
Note: There is some performance penalty associated with assembly bind failure logging.
To turn this feature off, remove the registry value [HKLM\Software\Microsoft\Fusion!EnableLog].

The part I didn't understand was the reference to Microsoft.VisualStudio.QualityTools.UnitTestFramework because that has nothing to do with NUnit - that is the MSTest library and I had no MSTest unit tests in my project at all.

That's when I realised this was a Kentico issue - because the Kentico CMS.Tests.dll that is referenced to help write fake Info objects and providers works for both NUnit and MSTest.

On my development machine Microsoft.VisualStudio.QualityTools.UnitTestFramework is installed with Visual Studio, as explained in this question, meaning every time I run the tests there is no issue - the dependent DLL can be found on my system. However, this is never available on a build server unless explicitly installed.

Solution

To solve this I followed this advice:

  1. Copied Microsoft.VisualStudio.QualityTools.UnitTestFramework.dll into my solution and committed it to my repo.
  2. Include a reference to this assembly in all my test projects, making sure Copy Local is true.

By simply referencing this DLL the build will copy it to the bin folder along with all other dependencies and can then be used in an environment where it is not globally available.

Peter Monks
  • 4,219
  • 2
  • 22
  • 38