12

I was under the impression that the test methods in a unit test class would be executed in the order that they appear in the class file. Apparently this is not true. It also doesn't appear to be purely based off of alphabetical order either. How does MSTEST decide execution order?

EDIT: I was able to track down the answer after digging a bit. See below.

Daniel Auger
  • 12,535
  • 5
  • 52
  • 73
  • F.Y.I.: "xUnit.net runs tests in random order", Ade Miller http://www.ademiller.com/blogs/tech/2007/11/xunitnet-runs-tests-in-random-order/ – gerryLowry Feb 15 '10 at 15:29

7 Answers7

12

I was able to track down the answer.

According to Microsoft employee Guillermo Serrato:

MSTest executes all tests synchronously, the order is nondeterministic

Daniel Auger
  • 12,535
  • 5
  • 52
  • 73
2

In my experiences (I was little hard) I tried a lot of thinks about the change order of the method execution orders in the MStest project. I'm going to try describe the steps.

  1. You should ensure delete the default .testrunconfig and .vsmdi files. These files createdby Visual Studio when the creating test project.

  2. Close the Visual Studio instance and reopen it.

  3. When the Visual Studio opened you should open the Test List Edior then Create your own list under the Lists of tests section. After you create a new test list it should be empty.

  4. Go to All Loaded tests section then drag any test onto newly created test section with your choised order.

Then ensure your last created test section checked and you can press the run test and voala it's running with your defined order.

I don't know this might bug or some thing like that, but it magically working. We tested this method a couple of time with my colleague.

Good lock

fyasar
  • 3,996
  • 2
  • 42
  • 55
2

Sorting in Test View, or test list editor causes them to appear like you have control -- it's just a by product of the implementation. We make no attempt to actually execute them in a specific order (We've gone back and forth on the "allow order", and "randomly order").

If you really need order, that is what ordered tests are for. These are available in all editions where Unit Test is available -- either use the Test/New Test menu, or right click the test project and create a "ordered test"

Dominic Hopton
  • 7,262
  • 1
  • 22
  • 29
2

It's very simple. I am using this

C:>mstest /testcontainer:C:\MyTest.dll /resultsfile:C:\MyTestresults.xml

The logic is here that by default mstest gives you a .trx file. So in the command against [/resultfile:] option write the name of file by giveing .xml extension. So I used /resultsfile:C:\MyTestresults.xml instead of /resultsfile:C:\MyTestresults.trx

This works fine for me. Let me know if it works for you.

2

Problem: What I want to do is to run only the web tests that are included into my ordered test. I'm using an ordered test just for that purpose, so I can run my web tests in the specific order I need them to run in. I have lots of web tests, but my ordered test contains a sub-set of those web tests, which I would like to run. Solution: An ordered test is used to run a set of tests in a certain order. Create an Orderer test list in Visual Studio/test project. From test view window (Test | Windows | Test View), select the OrderedTest and run it from there (you can right-click it and select "Run Selection"). The orderedtest will automatically run all your webtests that are included in it.

You will the the results in the test results window (Test | Windows | Test Results); double click the OrderedTest, and you will see detailed results for everyone of your webtests there. You can double click each webtest, and you will be able to see detailed results for each of them.

littleinstein
  • 61
  • 6
1

There are many ways to order the tests in VS. Use the test view and the add extra columns and order. I use VSMDI files and this runs them in order specified therein.

Preet Sangha
  • 64,563
  • 18
  • 145
  • 216
-1

It shouldn't matter what order they run in ... if your tests depend on another test running first, your tests are flawed :-)

Joel Martinez
  • 46,929
  • 26
  • 130
  • 185
  • 1
    Agreed - but what about the scenario when you are trying to track which test is not cleaning up after itself (in a data driven test) - knowing the execution order would help in this case – Delaney Feb 19 '10 at 05:49
  • I know this isn't always possible, I have a few test harnesses that are data driven in this way. But you should avoid data driven tests like this at all costs. Unit tests should only test units. Data driven tests should be integration tests, and be run using some other tool (ie. not mstest or nunit) where you can control execution order for exactly the reason you're describing :-) – Joel Martinez Feb 19 '10 at 19:11
  • agree, this has come up with testing a CRM, where it may be difficult to run tests against a 'fresh' set of data per test. But what if I have 100 ordered tests and there is a problem with test #93?! – DevDave Mar 27 '13 at 11:07
  • You can always just write your own testing framework that operates as you require :P I wrote my own unit testing framework that can be deployed to run in the target environment (ie. I originally wrote it to test XNA games on XBox) so that I could integration test things that require the graphics device: https://github.com/joelmartinez/scurvy.test – Joel Martinez Mar 27 '13 at 15:12