24

edit: note, question 288805 is similar, however, I specifically am asking how does MSTest choose the default test order. Please see the rest of this question. Thank you Eilon for the link.

I was looking at a legacy MSTest project. The tests were always running in the same order. The order was not alphabetic and was bouncing between methods in two *.cs TestMethod files.

I did not change the physical order of the legacy code. I did for my convenience append "MSTest01" to the method name of the first test, "MSTest02" to the method name of the second test, et cetera.

To my surprise, the execution order of the TestMethod functions changed; #3 first, #6 second, #5 third, et cetera.

When I removed the "MSTestnn" strings from the TestMethod function names, their execution order changed back to the previous ordering, i.e., one test from the first .cs file, two tests from the second .cs file, five tests from the first .cs file, et cetera.

It seems that file location may not be a factor while TestMethod function name may be a factor.

QUESTION: can anyone explain how MSTest decides on execution order of TestMethod functions?

Community
  • 1
  • 1
gerryLowry
  • 2,626
  • 5
  • 35
  • 44
  • 4
    Typically you should not care in what order your unit tests run. That being said, it is an intriguing question. – Mathias Feb 12 '10 at 21:49
  • 1
    @ Mathias I agree. I like the way xUnit.net runs tests in random order. Random order is the best choice because it is likely to reveal sequencing dependencies that should not exist. See my comments to Troy. – gerryLowry Feb 12 '10 at 22:16
  • ? weird, my comments to Troy seem to have disappeared, along with his question. – gerryLowry Feb 15 '10 at 15:26
  • http://www.ademiller.com/blogs/tech/2007/11/xunitnet-runs-tests-in-random-order/ F.Y.I. "xUnit.net runs tests in random order", Ade Miller – gerryLowry Feb 15 '10 at 15:27
  • Exact dup of: http://stackoverflow.com/questions/288805/how-does-mstest-visual-studio-2008-team-test-decide-test-method-execution-order – Eilon Feb 13 '10 at 02:31

4 Answers4

9

I believe that MSTest executes test methods ordering them by their 'ID' (seems to be their full namespace).

I created a TestProject1 which contains 4 unit tests (UnitTest1, ...2, ...A, ...B). Each unit test contains 5 test methods (TestMethodA, ...B, ...1, ...2, ...3). They were declared with random order inside their test classes. Now, every time I run MSTest, the tests are executed with the same order:

TestProject1.UnitTest1.TestMethod1
TestProject1.UnitTest1.TestMethod2
TestProject1.UnitTest1.TestMethod3
TestProject1.UnitTest1.TestMethodA
TestProject1.UnitTest1.TestMethodB
TestProject1.UnitTest2.TestMethod1
TestProject1.UnitTest2.TestMethod2
TestProject1.UnitTest2.TestMethod3
TestProject1.UnitTest2.TestMethodA
TestProject1.UnitTest2.TestMethodB
TestProject1.UnitTestA.TestMethod1
TestProject1.UnitTestA.TestMethod2
TestProject1.UnitTestA.TestMethod3
TestProject1.UnitTestA.TestMethodA
TestProject1.UnitTestA.TestMethodB
TestProject1.UnitTestB.TestMethod1
TestProject1.UnitTestB.TestMethod2
TestProject1.UnitTestB.TestMethod3
TestProject1.UnitTestB.TestMethodA
TestProject1.UnitTestB.TestMethodB

The only way to change that order is to rename one TestClass or a TestMethod. If for example I rename the TestMethodB, of the UnitTest1, to TestMethod4 it will be executed before TestMethodA.

To see the IDs of your test methods open the 'Test View' window from VS and then right click on a column header (e.g. Test Name) --> "Add/Remove Columns..." and add 'ID' column.

Bondolin
  • 2,793
  • 7
  • 34
  • 62
chaliasos
  • 9,659
  • 7
  • 50
  • 87
  • 6
    This behaviour is not specified and it can change without notice. – Dejan Dakić Aug 11 '14 at 08:14
  • How did you get MSTest to output the actual execution order? I'm am trying to achieve the same here: https://stackoverflow.com/questions/48379363/how-to-output-the-execution-order-of-unit-tests-in-visual-studio-2017 – philreed Jan 22 '18 at 11:06
4

As for VSTest execution order. Here is how it's organized in your TestProject:

  1. Sort cs-files in your project by their CREATION Time ASC
  2. Method Position in each file

For example, you have 3 cs files in project.

  • UnitTest1.cs - created 01/01/1970 with methods TestMethod05 and TestMethod03
  • UnitTest2.cs - created 05/01/1970 with method TestMethod02.
  • UnitTest3.cs - created 03/01/1970 with method TestMethod01.

Then order of executing test is this:

    TestProject1.UnitTest1.TestMethod05
    TestProject1.UnitTest1.TestMethod03
    TestProject1.UnitTest3.TestMethod01
    TestProject1.UnitTest2.TestMethod02

You can see the 'default order' using command:

vstest.console.exe TestProject1.dll /ListTests

Ivan Tsyng
  • 76
  • 4
  • Is there a way to change that default order in vstest, other that try to mess with file creation time(delete, recreate)? – Razkar Dec 07 '17 at 21:34
  • 1
    You can try to Use Playlist. Right click on the Test method in Test Explorer => Add to playlist => New playlist – Ivan Tsyng Dec 08 '17 at 09:34
0

From Microsoft's Document

For NUnit: Order by priority

 [Test, Order(5)]

For MSTest: With MSTest, tests are automatically ordered by their test name.

Note: Also you can use custom playlists

Fatih
  • 945
  • 15
  • 26
-2

The tests created in MSTest are ordered by execution time, in ascending order, after the tests that have not failed, for this reason the random execution. Unless the method names are in alphabetical order.

Wagner Ediel
  • 1