2

I'm experiencing an issue where tests are succeeding on my local machine using visual studio but they will fail when my CI Build runs them on my TFS server.

Here's the line that is failing and the preceding line for context.

var mockEventRepository = Substitute.For<IEventRepository>();
mockEventRepository.GetAll().ReturnsForAnyArgs(r => new List<Event>().AsQueryable());

Note that there are NO ARGUMENT MATCHERS! Not a single one in the entire test.

TFS will give the error:

NSubstitute.Exceptions.UnexpectedArgumentMatcherException : Argument matchers (Arg.Is, Arg.Any) should only be used in place of member arguments. Do not use in a Returns() statement or anywhere else outside of a member call.

In a nutshell the errors seem to be consistent (the same error) when they're occurring, but they also will seemingly randomly occur or not occur.

For instance... It just succeeded after queuing a subsequent identical build after the original one failed. That time however it failed when it tried the same build with a different configuration with a different but similar and erroneous error:

NSubstitute.Exceptions.AmbiguousArgumentsException : Cannot determine argument specifications to use.

FYI Using TFS 2015 and VS 2015.

Doug Morrow
  • 1,306
  • 1
  • 10
  • 18
  • 1
    Is it possible that there are arg matchers used in the tests running immediately before this one? If there are some being queued up but unused they can affect subsequent tests. The intermittency may then be due to tests being run in different orders. – David Tchepak Nov 11 '15 at 06:41
  • Thanks for the insight David. There's nothing shared between tests so unless NSubstitute caches its proxies I don't know if that's the case. I do have a strong feeling it test execution order that has something to do with it though. I actually did now manage to get it to fail in VS it just seems to succeed more often than fail on VS and fail more often than succeed in TFS. – Doug Morrow Nov 12 '15 at 17:37
  • I actually have a suspicion the real error is the AmbiguousArgumentsException and that it's caused by not using the 2 generic parameter method when mocking AutoMapper using IMappingEngine. Working on some modifications, will post answer if it turns out to be the fix! – Doug Morrow Nov 12 '15 at 17:53
  • Well I changed all of the AutoMapper subs to use ReturnsForAll because it was the only acceptable way to get rid of the ambiguity without rewriting the application to stop leveraging generic type inference. While this seemed to fix the AmbiguousArgumentExceptions it still randomly failed regarding argument matchers. – Doug Morrow Nov 12 '15 at 20:56
  • Then I paid close attention to the order of execution and discovered that it did seem to matter. I changed a Returns statement to ReturnsForAnyArgs and removed the argument matcher and now I can't seem to get any of the tests to fail anymore, although I don't think I've seen the same failing order of execution again... – Doug Morrow Nov 12 '15 at 20:57
  • At this point it seems to be working, but I'm rather unsatisfied with the result. I may choose to switch frameworks at this point, even though I prefer NSubstitute's syntactic sugar. – Doug Morrow Nov 12 '15 at 20:58
  • Yes need to make sure the generic call is for the right type. Also params/optional args can cause issues. NSubstitute's syntax definitely has a cost. Maybe try FakeItEasy or Moq. – David Tchepak Nov 12 '15 at 23:32
  • Thanks for all your help both here and on the mailing lists you monitor for NSubstitute David. I was able to finally discover I had missed a spot in my review of Arg matcher usages. I wonder how we can get more accurate errors in such a scenario? Based on my understanding, the system would have to detect that the arg matcher was invalid while adding it to the collection of matchers and that sounds exceedingly difficult to do. – Doug Morrow Nov 14 '15 at 01:00

2 Answers2

3

In a nutshell the entire issue was caused by an old test that had an Arg matcher in it by mistake without using NSubstitute.

The lesson that I learned here is that this type of scenario can produce very inaccurate errors and I would urge anyone experiencing a problem like this to review all uses of Arg matchers.

Doug Morrow
  • 1,306
  • 1
  • 10
  • 18
1

I was seeing this issue when using NSubstitute with xunit which by default runs test classes in parallel. There must be threading issues with NSubstitute because turning off parallelization fixed it for us. In your Test project AssemblyInfo.cs add

[assembly: CollectionBehavior(DisableTestParallelization = true)]

https://stackoverflow.com/a/34876963/1128742

patrickbadley
  • 2,510
  • 2
  • 29
  • 30