31

I have the latest NUnit(3.2.0) installed and I have all my tests run in parallel. It might look like desirable behavior but I didn't ask for it and actually it broke some of my tests. I have some initialization in [OneTimeSetUp] which is thread-dependent and it seems I can't do anything to force NUnit to run my tests sequentially. I've read the documentation and it states that by default tests aren't run in parallel but in fact they are!

Moreover, I've tried to add the following attribute: [assembly: Parallelizable(ParallelScope.None)] — no luck.

Does anybody know how to change this behavior?

P.S. I run it with ReSharper but also tried with MSVS add-in.

UPD: I'm using MVVM Light DispatcherHelper.Initialize()(inside[OneTimeSetUp]) to store the dispatcher object which is later used by a couple of tests. If threads are different(between a test and the setup method) then the action under test gets executed asynchronously and my tests fail.

I've checked the thread ids in different tests and they all are different.

UPD2: Excerpt from the documentation:

The NUnit 3.0 framework can run tests in parallel within an assembly. This is a completely separate facility from Engine Parallel Test Execution, although it is possible to use both in the same test run.

By default, no parallel execution takes place. Attributes are used to indicate which tests may run in parallel and how they relate to other tests.

If it doesn't mean the tests within an assembly should not be run in parallel until explicitly specified then what does it mean? And why [assembly: Parallelizable(ParallelScope.None)] has no effect on the tests parallel execution?

UPD3: Answer to the question might be found below but if you are stuck(as I was) with the DispatcherHelper.Initialize() you just need to remove this initialization from the OneTimeSetUp and put the following lines in every test that uses a dispatcher:

DispatcherHelper.Reset();
DispatcherHelper.Initialize();
ixSci
  • 13,100
  • 5
  • 45
  • 79
  • Your tests might not be running in parallel, it is just likely that they are running in a different thread than your OneTimeSetup. Are you storing information in thread local storage? – Rob Prouse Mar 14 '16 at 13:46
  • Please explain what you mean by "thread-dependent. Nothing in NUnit guarantees that tests run on the same thread. – Charlie Mar 14 '16 at 14:04
  • @RobProuse, updated the question – ixSci Mar 14 '16 at 16:03
  • @Charlie, I've update the question. Also, doesn't the documentation states that the tests should run in one thread(not parallel) in the default case? – ixSci Mar 14 '16 at 16:05
  • I don't see that in the docs anywhere, but I can imagine how you might make that assumption. In going from 3.0 to 3.2, the internal implementation changed, so if you were happy on 3.0.1, you might try going back to it. – Charlie Mar 14 '16 at 19:26
  • Also, if you or anyone has an idea as to how we might express the need for a single thread using an attribute, please file a feature request (issue) on github! – Charlie Mar 14 '16 at 19:27
  • 1
    The documentation only states that tests will run sequentially or in parallel. You may construe that this means they run on the same thread, but there are many reasons that the internal implementation might require tests to run on different tests. Timeout is an example, where we spawn a thread and kill it if the test times out, but there are many others. – Rob Prouse Mar 14 '16 at 19:49
  • @Charlie, could you please comment on the latest post update? – ixSci Mar 15 '16 at 05:23
  • @RobProuse, is it what happens under the hood in NUnit or it is just a guess? If it is the the former then please extract it to the answer and I will accept it. – ixSci Mar 15 '16 at 05:28
  • 1
    @ixSci, yes I know that is what happens under the hood, I am one of the team owners. There was a PR between 3.0.1 and 3.2 that caused more tests to run on their own threads to solve some bugs with things like timeout and apartmentstate, but even before that, some tests would have run on different threads. You probably just got lucky before. Sorry... – Rob Prouse Mar 15 '16 at 21:51
  • 1
    Commenting on the update, as requested by @iXsci: the docs you quote are correct. NUnit will not run tests in parallel without your sayso. That doesn't mean it will run them all on the same thread. It may use separate threads but will run those threads sequentially. – Charlie Mar 16 '16 at 19:00
  • @ixSci, I have updated my answer with links to the NUnit issue and pull request. We will change the behaviour to allow you demand a single thread. Also, for your UPD3, can't you add that code to a SetUp method? I believe they run on the same thread as their test. – Rob Prouse Mar 18 '16 at 17:27

2 Answers2

27

NUnit does not guarantee that all of your tests will run on the same thread, so the observation that your tests are running on different threads does not mean they are running in parallel.

The documentation only states that tests will run sequentially or in parallel. You may construe that this means they run on the same thread, but there are many reasons that the internal implementation might require tests to run on different threads. Timeout is an example, where we spawn a thread and kill it if the test times out, but there are many others.

Parallel test runs are new to NUnit 3, so the internal implementation changed from NUnit 2. An attribute that forces all tests within a thread to run on the same thread might be useful, so feel free to submit an enhancement request.

Sorry, I am unfamiliar with MVVM Light, so I can't suggest ways to marshal back to the OneTimeSetup thread.

Update - Since this is a common usage with web and async, the NUnit team has decided to provide an attribute that will demand tests be run on the same thread as the fixture's OneTimeSetup. This will be in the next release, either 3.4, or in a hotfix 3.2.1 release. If you want to track progress, see the issue and the pull request.

Update 2 - You can now add SingleThreadedAttribute to a TestFixture to indicate to the runner that the OneTimeSetUp, OneTimeTearDown and all the child tests must run on the same thread.

TN.
  • 18,874
  • 30
  • 99
  • 157
Rob Prouse
  • 22,161
  • 4
  • 69
  • 89
  • 3
    Thanks for the answer. Don't you think it would be good to mention that tests might be run on the different threads even in a non-parallel fashion? In some prominent place, I think. This answer should help also, because now it would be easy to google such a behavior but I also think that the documentation should somehow mention it. – ixSci Mar 16 '16 at 05:10
  • It's hard to write this kind of documentation in advance because it's hard to know what assumptions may be made by users. So now we see this one should join others we already knew about. :-) – Charlie Mar 16 '16 at 17:14
  • 5
    I may add a docs page called "Things You Must Not Assume" :-) For now, here are a bunch of false assumptions I have come across over the years: (1) Don't assume your constructor is only called once per run by NUnit (2) Don't assume your test is running on a unique thread - although you can ask NUnit to do that (3) Don't assume your tests are all run on the same thread (4) Don't assume your tests are all run in the order they appear in the source file or in alphabetical order. Seriously, please post suggestions for such a page on github, https:github.com/nunit/docs – Charlie Mar 16 '16 at 19:08
26

You can prevent tests from running in parallel by adding the [NonParallelizable] attribute, which can be added in test, class and assembly level.

bytedev
  • 8,252
  • 4
  • 48
  • 56
Nir
  • 1,836
  • 23
  • 26