1

I'm trying to get a bunch of MSpec tests to run on multiple cores in TFS 2013. It doesn't appear to do it out of the box. It can run MSpec, but only in sequence and it takes a over an hour.

I am following this guide, but in step 4 he says replace the Foreach Xaml element with ParallelForEach to get the tests to run in parallel. I downloaded the default build template in TFS 2013. It is a lot simpler, but it doesn't have this tag.

It has:

<mtba:RunAgileTestRunner 
  DisplayName="Run VS Test Runner" 
  Enabled="[Not AdvancedTestSettings.GetValue(Of Boolean(&quot;DisableTests&quot;, false)]"
  TestSpecs="[AutomatedTests]"
  ConfigurationsToTest="[ConfigurationsToBuild]" />
Anthony Mastrean
  • 21,850
  • 21
  • 110
  • 188
Crab Bucket
  • 6,219
  • 8
  • 38
  • 73
  • If you switch to TeamCity you should be able to run the MSpec tests in parallel see: http://stackoverflow.com/questions/24611933/how-do-i-run-mspec-test-assemblies-in-parallel – ds19 Mar 04 '15 at 11:25
  • @ds19 Thanks but moving to TeamCity is not an option for us. The solution must lie within TFS – Crab Bucket Mar 04 '15 at 11:52
  • I don't understand how TFS is running MSpec tests at all... that `mtba:RunAgileTestRunner` doesn't seem to support MSpec. Can you show your existing test task? The one that runs in sequence? – Anthony Mastrean Mar 05 '15 at 16:29
  • @AnthonyMastrean the way we were integrating MSpecs was using the MSpec visual studio test adapter as here http://stackoverflow.com/questions/26871402/mspec-test-adapter-with-tfs but this only seems appropriate for relatively few tests – Crab Bucket Mar 10 '15 at 15:15
  • Oh, that doesn't sound good. I've not used the TFS build system. It should be able to shell-exec and read the resulting XML/HTML report. – Anthony Mastrean Mar 10 '15 at 15:40

2 Answers2

2

The default MSpec test runner cannot run tests in parallel. That's why you see the reimplementation of a parallel test runner.

I doubt that TFS is implementing an MSpec test runner from the framework source code (although that would be possible). That parallel test runner is using internal classes, like ISpecificationRunner, and running them in parallel.

Your only options, if you must stick with MSpec and TFS, are

  • Split your tests into multiple projects/assemblies and feed them to a TFS parallel task that shell-executes the default test runner

  • Use a TFS shell-execute task to run your tests through the parallel runner

Anthony Mastrean
  • 21,850
  • 21
  • 110
  • 188
  • This is almost what we did. We basically run the tests through the command line with the post script build path property in the default template. It doesn't parallel it (so not like this) but the time to run the tests drop from 77 minutes to 8. We lose all reporting but at least failed tests will break the build. The flaw is the poor performance with the VS MSpec Test Adapter - it hemorrhages memory so hits the 4GB limit on a 32 bit process very rapidly. The console runner (or Resharper) runner does a lot better so can be used with the build – Crab Bucket Mar 10 '15 at 15:19
0

I am assuming that if you want to run tests in parallel they are integration tests that take a long time to run.

If that is the cas then you should move all non Unit Tests out of the build and push them further down the pipeline.

http://nakedalm.com/execute-tests-release-management-visual-studio-2013/

You can use Release Management to deploy your application and run your integration tests. Here you can run a larger number of long running tests without locking your build servers.

MrHinsh - Martin Hinshelwood
  • 23,409
  • 5
  • 46
  • 69
  • Thanks for your response. However moving the tests out of the build is not an option. Running the build on a less frequent schedule is (2 hourly) but the performance problems with the tests is so severe that it often crashes the App Domain. The position we are in is not good and a review of the general test strategy is due - however we want these tests under TFS build in the meantime – Crab Bucket Mar 04 '15 at 10:22