3

I have tests in two separate classes within a MSTest project, each class is (I think) set up properly and each class runs fine but when I run the whole project's tests, some fail.

My two classes both involve setting up some external data but each class is designed to make sure this is in a known state before starting (and to wipe it after finishing). I would have though MSTest might run all methods in a test class in parallel(?), but would run each class in sequence... is this an incorrect assumption?

Edit: I came across this question (how does MSTest determine the order in which to run test methods?) It seems to suggest VS may interleave tests from multiple classes in a (seemingly but not actually) random order i.e. it does not run all tests in classA before starting in classB. This is problematic in my case because even if I order my tests within a class, MSTest might still run methods from multiple classes which conflict?

Community
  • 1
  • 1
Mr. Boy
  • 60,845
  • 93
  • 320
  • 589
  • http://stackoverflow.com/questions/3917060/how-to-run-unit-tests-mstest-in-parallel – L-Four May 09 '16 at 11:55
  • @L-Three do I understand it from that, that VS does _Not_ run tests in parallel unless I explicitly ask it to? – Mr. Boy May 09 '16 at 11:59
  • Indeed, sequentially by default. As from VS 2010, in parallel is possible. Also read https://blogs.msdn.microsoft.com/vstsqualitytools/2009/12/01/executing-unit-tests-in-parallel-on-a-multi-cpucore-machine/. – L-Four May 09 '16 at 12:02
  • 2
    Rule: each test should be completely independent of any other test; and order of execution should not matter. – L-Four May 09 '16 at 12:07
  • I can do this if each method does the full setup/teardown functionality instead of doing it in class init/teardown, that just seems a bit inefficient. Partly I just didn't expect different class' tests would be interleaved _because_ we have class setup/teardown functionality which _can_ affect the environment – Mr. Boy May 09 '16 at 12:12
  • Take a look at this answer I gave on the order of tests. To get the behavior you want, you'll probably have to use ordered tests. http://stackoverflow.com/a/24131990/573218 – John Koerner May 09 '16 at 14:55
  • I am having a similar problem -- MSTest seems to be running multiple tests at the same time, and I do not have parallelism turned on. – ryanwebjackson Sep 20 '18 at 17:58

1 Answers1

3

Important note

MsTest doesn't guarantee the order of execution and order can be different between runs. If you need to rely on order you'll need to created an Ordered Test, a VS Premium/Enterprise feature unless you're on Visual Studio 2015 update 2, which brought them to Pro.

Test execution can interleave, bounce and do all kinds of crazy things. Though on some versions of MsTest it tends to run in alphabetical order of the full namespace of the test, e.g.: my.namespace.class.method.

When using data driven tests it's even weirder as the order of the data coming from the datasource acts as an additional randomizer.

Do not rely on order It's probably better to use TestInitialize and run your code before each test executes than to rely on [Class|Assembly]Initialize to setup your data in ways that are incompatible between different tests.

Visual Studio 2015 and earlier.

MsTest can execute tests in parallel using the legacy test runner and when specifying parallelism in the testSettings file. In the legacy runner tests will be executed on up to 5 threads. The legacy test runner will not constrain the tests in any way. It will run all tests in parallel if it can and there is no specific order or protection. You can see in the screenshot below:

enter image description here

Unless your testSettings explicitly mention the parallism, the tests will run sequentially, but with no specific order. I suspect it will use the IL order of the compiled assembly or alphabetical order. The order in the source file is definitely not what is used.

In Visual Studio 2015 update 1 and later

Visual Studio 2015 update 1 introduced the option to run tests in parallel in the new test runner. The new test runner uses a different approach and will parallellize the tests per container. For C# this means this means that tests in one assembly are executed sequentially while tests that are in separate assemblies can be executed in parallel.

It is assumed that Integration tests and UI tests are kept in their own separate solution and that when you enable this parallel option you have a solution containing only Unit tests.

jessehouwing
  • 106,458
  • 22
  • 256
  • 341
  • Thanks. From the comments so far I realise this isn't a parallelism question but an interleaving/sequential quesiton, see updated title – Mr. Boy May 09 '16 at 12:06
  • Yeah, there is no explicit order in test projects, unless your classes both have a `ClassInitialize` attribute as far as I know, in that case the tests from these two classes won't be interleaved. – jessehouwing May 09 '16 at 12:11
  • Are you saying that classes employing `ClassInitialize` are guaranteed to have all their tests run uninterrupted? Because that side-comment would basically resolve my issue – Mr. Boy May 09 '16 at 12:19
  • Also in VS2013 I cannot find a way to see what order the tests _are_ run in. Test Explorer doesn't seem to show it – Mr. Boy May 09 '16 at 12:27
  • I'm saying I remember that ClassInitialize would act as a boundary between tests, but it's a long time I wrote MsTest and again there's very little documentation. I'd say try it. – jessehouwing May 09 '16 at 12:37