8

What's the sequence of events in a full MSTest run of unit tests in C# inside Visual Studio (Ctrl+R, A)?

Here's what I think so far:

  • 1 - Runs [AssemblyInitialize]
  • 2 - Randomly runs a [ClassInitialize]
  • 3 - Runs the class [TestInitialize]
  • 4 - Randomly runs a [TestMethod] from that class
  • 5 - Runs the class [TestCleanup]
  • Repeat 3 through 5 for each TestMethod in the class
  • Repeat 2 through 5 for each test class
  • 6 - Runs all classes [ClassCleanup] methods
  • 7 - Runs [AssemblyCleanup]

But I think VS might initialize multiple classes at once and then randomly run TestMethods. Should the tests be autonomous across its class or across the whole test project, or even the whole solution? Knowing the exact sequence of events should answer those questions.

UPDATE:

I did some tests and found that it is indeed the order in which events occurs, except for #3 to 5 where ANY test from ANY class could run. Visual Studio seems to sequentially run one test at a time. However, one should not rely on this for reasons explained in accepted answer.

Jerther
  • 5,558
  • 8
  • 40
  • 59
  • yea its definitely MSTest framework, did you read https://ict.ken.be/unit-testing-with-mstest ? – Dejan Dakić Aug 20 '14 at 13:47
  • If I'm not mistaken you can have test case ordering attribute, if you want to make sure the sequence is right. – Yuliam Chandra Aug 20 '14 at 13:49
  • Seriously? Can you provide a name of this attribute? I only know about test "Ordered Test", file.orderedtest where you specify the order (http://msdn.microsoft.com/en-us/library/ms182630(v=vs.90).aspx) – Dejan Dakić Aug 20 '14 at 13:51
  • yeah It'S for MSUnit, i'Ve updated the question. – Jerther Aug 20 '14 at 13:52
  • Actually it is MSTest not MSUnit :) – Dejan Dakić Aug 20 '14 at 13:53
  • Did you miss the part I said _If I'm not mistaken_, yes I just read it, it's not an attribute, it's in the test window – Yuliam Chandra Aug 20 '14 at 13:53
  • But without being ordered, the order isn't relevant to my question. I mean, there is a certain level of order, for example, the assemblyinitialize attribute gets run only once at the beginning. – Jerther Aug 20 '14 at 13:53
  • Jerther the rules you have written in question are correct. BUT depends on the test runner, for example: vstest.console does not guarantee TestCleanup will be called, and same goes for AssemblyCleanup. There is also similar rule with nCrunch. You can, altho, write your own runner... :) – Dejan Dakić Aug 20 '14 at 13:56
  • I've specified Visual Studio as the runner in the question. – Jerther Aug 20 '14 at 14:17
  • I am wondering when is the constructor of the Test Class run? That is when is the Test Class instantiated ? Is new instance of the class created before each unit test or is new instance of the class created for all unit tests from that class ? – Dejan Bogatinovski Nov 01 '15 at 03:38
  • 1
    @DejanBogatinovski, it may not be well known that a new instance of the class is created for each TestMethod ([related answer](https://stackoverflow.com/a/7079605/538763)) so the constructor would run before each test. – crokusek Jul 10 '19 at 00:31

2 Answers2

3

You are correct. This is indeed the order in which the code will be run. However, since tests should be completely independent, there are no guarantee that they will be run in order and that they will be run on a single thread. The framework could run multiple test at the same time.

You can force a specific test order through the use of test cases if you need too, but this is considered bad practice as test cases should be used to regroup tests together (tag them) instead.

Etienne Maheu
  • 3,235
  • 2
  • 18
  • 24
  • This would mean that if I need a database in my test, I cannot use the same table in two tests because they could run at the same time? Let'S say I add a row in a table and then count them, another test method could modify that same table and make my first test fail! – Jerther Aug 20 '14 at 14:08
  • This would only happen if your tests are badly designed. First, unit tests should test objects in isolation. Using a real database as a backend makes this an integration test. These tests should be kept in test cases in an MSTest point of view to ensure their run order. Second, MSTest can only run test classes in parallel. Not individual tests within a specific class. If two classes uses the same feature, maybe you should consider decoupling your code further. – Etienne Maheu Aug 20 '14 at 14:15
  • So two test classes could and WILL run at the same time, right? – Jerther Aug 20 '14 at 14:21
  • 1
    Could, not will. There is just no guarantee that they won't and some test runners (like dotCover) will run multiple tests at the same time depending on your configuration. – Etienne Maheu Aug 20 '14 at 14:29
  • 1
    Oh this complicates things... I knew tests had to be autonomous but I didn't know they could run at the same time. This answers my question, Merci! – Jerther Aug 20 '14 at 14:35
  • Like I said, if you really need the tests to be run in a specific order, you can always use test cases for this. The main drawback is that they are stored in separate files and you can only add tests in a test case, not entire classes or assemblies. As a result, if you add a new test, you need to remember to add it to the test case. So yeah... it's best to architecture your way out of these situations if you can. – Etienne Maheu Aug 20 '14 at 15:25
1

To define a specific order for your tests, either create an ordered test (http://msdn.microsoft.com/en-us/library/ms182631.aspx) or create a match file and call mstest.exe for each test case in the specified order you desire (http://msdn.microsoft.com/en-us/library/ms182489(VS.80).aspx)

Paully
  • 166
  • 2
  • 3