0

I'm running my unit tests on a 16-core machine. I can't see any difference in elapsed time when using no parallelization parameter, --workers=1 and --workers=32. I'd like to confirm that NUnit really is running the expected number of simultaneous tests, so that I don't spend time hunting down a non-existent problem in my test code.

I have [Parallelizable] (default scope, ParallelScope.Self) on the common base class. It is defined not on any other class or method. I'm using nunit3-console, both via Jenkins and on my local command line.

Is there a way to tell that tests are running in parallel? NUnit is reporting the correct number of worker threads, but there's no report saying (for example) how many tests were run in each thread.

Run Settings
    ProcessModel: Multiple
    RuntimeFramework: net-4.5
    WorkDirectory: C:\Jenkins\workspace\myproject
    NumberOfTestWorkers: 16

I can log the start and finish times of each test then manually check that there's a reasonable number of overlaps; is there any simpler and more repeatable way of getting what I want?

Paul Hicks
  • 13,289
  • 5
  • 51
  • 78
  • [Which pattern is nunit using](http://stackoverflow.com/questions/1784928/c-sharp-four-patterns-in-asynchronous-execution). [pnunit](http://www.nunit.org/index.php?p=pnunit&r=2.5) could be another option. Does cmd line tasklist contain the child processes of nunit? – lloyd Dec 02 '15 at 22:34
  • Isn't nunit3 essentially nunit2+pnunit (+ some excellent refactoring)? – Paul Hicks Dec 02 '15 at 22:59
  • The tests are totally independent (well, as much as web requests are). So "fire and forget" from the list in that question you linked to. – Paul Hicks Dec 02 '15 at 23:01

2 Answers2

0

I turned on --trace=Verbose which produces a few files. InternalTrace.<pid1>.log and InternalTrace.<pid2>.<dll_name>.log together contained a thorough description of what was happening during the tests. The per-agent log (the one with the DLL name in the log file name) was pretty clear about the state of parallelization.

16:13:40.701 Debug [10] WorkItemDispatcher: Directly executing test1
16:13:47.506 Debug [10] WorkItemDispatcher: Directly executing test2
16:13:52.847 Debug [10] WorkItemDispatcher: Directly executing test3
16:13:58.922 Debug [10] WorkItemDispatcher: Directly executing test4
16:14:04.492 Debug [10] WorkItemDispatcher: Directly executing test5("param1")
16:14:09.720 Debug [10] WorkItemDispatcher: Directly executing test5("param2")
16:14:14.618 Debug [10] WorkItemDispatcher: Directly executing test5("param3")

That third field looks like a thread ID to me. So I believe that the agent is running only one test at a time, even though (I think..) I've made all the tests parallelizable and they're all in different test fixtures.

Now I've just got to figure out what I've done wrong and why they're not running in parallel...

Paul Hicks
  • 13,289
  • 5
  • 51
  • 78
  • Hi Paul - looking at a similar problem now. Did you ever reach a conclusion on what may be wrong? – Chris Dec 22 '15 at 15:54
  • Yes. `[Parallelizable]` is not inheritable: see [Charlie Poole's comment](https://groups.google.com/forum/#!topic/nunit-discuss/aXduucHkMF0) on the NUnit-Discuss group. I took his advice and gave the assembly the attribute via AssemblyInfo.cs, then annotated the non-parallel tests with `[Apartment(ApartmentState.STA)]`. I can look for a question (or create one) to add this as an answer to, if that's useful. – Paul Hicks Dec 22 '15 at 20:31
  • Thanks Paul - I did a bit more digging this afternoon however, and I think mine was memory limited...after closing visual studio it started taking up to 4GB memory before it hit the next limit, and run times improved. Thanks for the debug details above however, that pointed me in the right direction. – Chris Dec 22 '15 at 20:41
0

I could be mistaken but it seems to me that to set [Parallelizable] parameter is not enough to make tests run in parallel. You also need to create nodes that will run tests. So if you use Jenkins you have to create Jenkins Slaves and only then your tests will be run in parallel.

So what I want to tell is that as I understand there is no possibility to run tests in parallel on one PC.

If there is such a possibility (to run tests in parallel on the same machine) it would be really great and I struggle to hear about it!

Denis Koreyba
  • 3,144
  • 1
  • 31
  • 49
  • No, it works fine if I add the Parallelizable attribute to each TestFixture. If we went so far as to use separate Jenkins jobs, then parallelization would not come into it. I could run non-parallel tests in parallel using that technique. – Paul Hicks Dec 03 '15 at 20:54