4

We are thinking about moving our tests from MSTest to XUnit.

Is there any migration application that takes a MSTest and migrates it to XUnit?

Also, if not, what should I look out for when doing this?

Thanks. JD.

JD.
  • 15,171
  • 21
  • 86
  • 159
  • How many? It helps a lot as the advice for 5000 tests in 500 files will be different to if you have <100. – Ruben Bartelink Nov 18 '09 at 09:06
  • It is just under 500 tests at the moment. – JD. Nov 18 '09 at 10:52
  • If not a sicret, what is the reason to move? – Mike Chaliy Nov 18 '09 at 11:35
  • @Mike, we do not want to install VS on the test server. That is the main reason. – JD. Nov 18 '09 at 14:28
  • @JD: That's a bad reason to move to NUnit IMO. And in 2010 there's going to be a standalone MSTest installer (https://connect.microsoft.com/VisualStudio/feedback/ViewFeedback.aspx?FeedbackID=505807&wa=wsignin1.0 http://blogs.msdn.com/jimlamb/archive/2008/11/03/integrating-dependencies-into-team-build-2010.aspx). One other reason you need MSTest on the server is if you're using TFS Test Publishing, see http://stackoverflow.com/questions/1062994/running-mstest-exe-publish-on-a-teambuild-server-what-are-the-prerequisites (This is my final problem, after Test References) – Ruben Bartelink Nov 18 '09 at 15:54
  • @Ruben: You have got me thinking. Maybe nUnit is not the way to go. RTM for VS2010 is March 2010. Do you think we could use the beta's MSTest before then? – JD. Nov 19 '09 at 10:28
  • @JD: Sorry only saw this now. Since then VS has slipped "a few weeks" which may influence your decision. But even Beta2 has a go-live license. I personally wouldnt install the Beta2 on my production CI server - I'd probably just install VS and live with it for now - then you can install RTM and get rid of that dirty feeling at that point. There's not much to be gained by putting beta bits on a production server. My main point is that if you are going to go through the pain of migrating tests, it'd better be for more than just a desire to remove dirtiness. – Ruben Bartelink Dec 21 '09 at 10:24
  • Have you considered the data loss you will experience as a result of the move? You will loose the ability to relate tests to requirements as well as the ability to report on trend data for code coverage and test results. If you are using Visual Studio 2010 you will also loose the use of Test Impact Analysis as well. I for one would not want to loose the ability to see which of the 40k Unit Tests I need to run to make sure I have covered at least the code changes that have been made! – MrHinsh - Martin Hinshelwood Mar 13 '11 at 22:38

2 Answers2

3

I moved quite a few tests recently. It depends on how many and what type of tests you're converting, and you didnt kill yourself giving us details. In general, I think its safe to assume that your average MSTest minded shop wont be massively Test Infected and thus wont have delved into each dark corner of MSTest.

All the Assert.* methods and the basic Test Attributes are simple find and replaces. The more rare ones, I'd generally steer towards assessing each case individually. Unless you're already a xUnit.net expert, you've got lots to learn and this will help you.

Also, usage of Assert.Fail isnt a simple transformation. The other thing is the transformation of TestClassInitialize to IUseFixture - simple to do, but hard to automate.

If people are using Test References, you won't be able to remove the reference to the MSTest assembly (and you'll still need to have VS on your build server - and it will continue to randomly fail on the Shadow taks, see my questions).

The biggest manual work for me was going through the 20 lines of boilerplate in a region at the top to see whether anyone actually used any of the custom attributes before deleting them.

The main thing that would have been a lot of work had it not been for a CodeRush template was converting ExpectedException to Assert.Throws. If you havent got CodeRush or ReSharper on this job, you'd be stealing money from your employer.

Ruben Bartelink
  • 59,778
  • 26
  • 187
  • 249
  • We have around 500 tests over about 200 files. I have not used XUnit before, I just saw some good reviews on it and the fact that XUnit is created by the NUnit developers, I thought that would be the way to go. Since I only wrote a few of the tests, I do not have all the details of all the other tests. – JD. Nov 18 '09 at 11:00
  • 2
    @JD: xUnit.net is definitely worth it in my experience - it makes you write better, shorter, cleaner tests. If you read Osherove's book and xUnit Test Patterns you'll appreciate it even more over the other frameworks. I've had only good experiences in my interactions with the team too (in terms of response to requests being sensible, timely etc.). 500 tests is in the realm of what I moved. What are you planning for runners etc.? @Others: Anyone else got porting experiences/pitfalls? – Ruben Bartelink Nov 18 '09 at 11:15
  • @Ruben. Thanks. The idea is to incorporate the tests into CruiseControl (msTest would have required installing VS and we did not want to do that). We have resharper on our development machines. I will definately look at xUnit over the next few weeks. Can you recommend any good tutorial/vids etc. Thanks once again. – JD. Nov 18 '09 at 12:43
  • 1
    @JD: Be sure you dont have Test References (Private Accessors) before you switch to NUnit if you dont want a VD dependency. I find the articles at http://xunit.codeplex.com/#blog to be the best learning resource - you'll find something that suits your learning style. In general, the source and the samples are enough though - they're small and neat. If you're budgeting more than a day for porting to NUnit, you need to figure out whether all you're doing is getting over an unclean feeling to install VS on your CI server- remember there's a separate 'MSTest' install package for CI servers in 2010 – Ruben Bartelink Nov 18 '09 at 13:17
  • @JD: Team System Test in 2005 introduced Private Accessors/Test References. This technology is referenced [IMO] in caps in http://www.codethinked.com/post/2009/11/05/Ite28099s-Okay-To-Write-Unit-Tests.aspx. It uses a MsBuild Task called Shadow which calls an exe called publicize.exe which produces an assembly containing ClassName_Acessor classes with methods that can invoke private methods in an assembly using reflection. If people have used that, you'll know by Find-in-Files for '_Accessor' or by seeing Test References folders in your test projects. – Ruben Bartelink Nov 18 '09 at 15:18
  • In my case, they were used so I sadly have VS on my build server (and publicize.exe regularly makes my build fail only to work when retried.) (TMI: I'm in the process of routing around this issue by using a Test Stripper) – Ruben Bartelink Nov 18 '09 at 15:20
1

Consider Compare MSTest and xUnit

Mike Chaliy
  • 25,801
  • 18
  • 67
  • 105
  • 1
    @JD - note that it would be simpler to move from MSTest to either NUnit or MbUnit, as Mr. Chaliy's link clearly shows. If your primary goal is to migrate away from MSTest, those may be better choices. If you agree with the xUnit.net philosophy, never mind. :) – TrueWill Nov 17 '09 at 18:33
  • @TrueWill: While I wouldn't be standing around saying "What's wrong with MSTest", I'd see a move to NUnit or MbUnit as a sideways move - at the end of the day MSTest has pretty much got a one-to-one mapping to NUnit and you havent gained much, except for lots of MbUnit Attributes (if you consider that a 'gain'). – Ruben Bartelink Nov 18 '09 at 09:01
  • Thanks guys, As I mentioned below, I have not used xUnit before and migrating 500 tests may be too much of a task. I will move to NUnit for now and then hopefully on new projects look into the xUnit philosphy. – JD. Nov 18 '09 at 11:03
  • @JD: You could give it a try on a subset and see how it goes. What are the key things you feel you're going to gain by porting to NUnit? IOW How do you justify the time on that porting process? To me MSTest and NUnit are not dramatically different (though in a straight fight I know which one I'd pick)? – Ruben Bartelink Nov 18 '09 at 11:24