NUnit vs. Visual Studio 2008's test projects for unit testing

Question

I am going to be starting up a new project at work and want to get into unit testing. We will be using Visual Studio 2008, C#, and the ASP.NET MVC stuff. I am looking at using either NUnit or the built-in test projects that Visual Studio 2008 has, but I am open to researching other suggestions. Is one system better than the other or perhaps easier to use/understand than the other?

I am looking to get this project set up as kind of the "best practice" for our development efforts going forward.

score 99 · Answer 1 · edited Jul 30 '20 at 11:22

99

Daok named all the pro's of Visual Studio 2008 test projects. Here are the pro's of NUnit.

NUnit has a mocking framework.
NUnit can be run outside of the IDE. This can be useful if you want to run tests on a non-Microsoft build server, like CruiseControl.NET.
NUnit has more versions coming out than visual studio. You don't have to wait years for a new version. And you don't have to install a new version of the IDE to get new features.
There are extensions being developed for NUnit, like row-tests, etc.
Visual Studio tests take a long time to start up for some reason. This is better in Visual Studio 2008, but it is still too slow for my taste. Quickly running a test to see if you didn't break something can take too long. NUnit with something like Testdriven.Net to run tests from the IDE is actually much faster. Especially when running single tests. According to Kjetil Klaussen, this is caused by the Visual Studio testrunner. Running MSTest tests in TestDriven.Net makes MSTest performance comparable to NUnit.

edited Jul 30 '20 at 11:22

Peter Mortensen

30,738
21
105
131

answered Sep 18 '08 at 14:33

Mendelt

36,795
6
74
97

Also I believe VS unit testing is not available in the Professional version of Visual Studio (at least it wasn't with VS2005). The MS Patterns and Practices group provide both MS and NUnit versions of their unit tests, so they obviously recognize this as a problem for many of us poorer users! – Joe Sep 20 '08 at 14:35
I seem to remember that this has changed for 2008. I'm not sure though. My boss gave me the team edition with all kinds of features to ignore :-) – Mendelt Sep 20 '08 at 19:41
From my experience, the slow performance is caused by the testrunner in VS - not the testing framework itself. You can use TestDriven.Net on MSTest as well, and the performance is comparable to NUnit as far as I can tell. – Kjetil Klaussen Feb 05 '09 at 11:57
@Kjetil Klaussen: That's useful information, I'll update my answer. – Mendelt Feb 05 '09 at 12:28
19

Can't you just use mstest.exe to run MSTest tests outside the IDE? – Phillip Wells Apr 29 '09 at 20:26
13

Nunit having a mocking framework is not much of an advantage. I've been using VS 2008 unit test projects with the Moq framework which uses a novel approach, leveraging LINQ expression trees: http://code.google.com/p/moq/ – dso May 27 '09 at 19:49
5

One more plus for Nunit, for me anyway, is that Resharper has a very nice UI wrapped around it which is much faster than the VS testing components. It even hyperlinks the stack trace of failed tests to your code. – Jeff Putz Jul 27 '09 at 00:00
@DSO - FYI, Rhino Mocks 3.5 also supports expression trees. – Richard Szalay Jul 27 '09 at 14:30
7

Unit testing is included in the professional version of VS 2008. – user179700 Oct 05 '09 at 22:20
3

@Jeff Putz: Resharper can run the Visual Studio unit-tests too, even outside of a test project. – Paul Ruane Aug 03 '10 at 15:38
NUnit has Test and Fixture level setup and teardown and row based testing with TestCase... Plus it has an xml output file that can be aggregated into CC.net – JDPeckham Mar 28 '11 at 02:28

score 64 · Answer 2 · edited Jul 30 '20 at 11:25

64

The unit-testing framework doesn't actually matter much, because you can convert test classes with separate project files and conditional compilation (like this, Visual Studio → NUnit):

 #if !NUNIT
  using Microsoft.VisualStudio.TestTools.UnitTesting;
 #else
  using NUnit.Framework;
  using TestClass = NUnit.Framework.TestFixtureAttribute;
  using TestMethod = NUnit.Framework.TestAttribute;
  using TestInitialize = NUnit.Framework.SetUpAttribute;
  using TestCleanup = NUnit.Framework.TearDownAttribute;
  using TestContext = System.String;
  using DeploymentItem = NUnit.Framework.DescriptionAttribute;
 #endif

The TestDriven.Net plugin is nice and not very expensive... With only plain Visual Studio 2008 you have to find the test from your test class or test list. With TestDriven.Net you can run your test directly from the class that you are testing. After all, unit tests should be easy to maintain and near the developer.

edited Jul 30 '20 at 11:25

Peter Mortensen

30,738
21
105
131

answered Sep 18 '08 at 14:59

Tuomas Hietanen

4,650
2
35
43

12

I voted this one down because NUnit has a richer syntax than MSTest, which means you can go from MSTest -> NUnit, but not vice versa unless you are VERY careful. History shows at least one of us aren't. – Thomas Eyde Feb 09 '09 at 11:43
2

I agree with Thomas. This will work assuming you are using the most basic assert statements, but the NUnit constraint model is very powerful and reason enough to choose NUnit over MSTest. – Mark Apr 23 '09 at 13:57
I believe, this approach is taken by the ms pattern and practice group in the EntLib tests. – robi-y May 17 '09 at 19:54
There are significant limitations going from MSTest -> NUnit as well. MSTest has constructs (eg PrivateObject) to access private members of a class to test; NUnit does not. – Dan Is Fiddling By Firelight Jan 07 '10 at 13:58
1

@Dan Neely if you're testing privates you're doing it wrong :( – JDPeckham Mar 28 '11 at 02:30
@JDPeckham When you have 100s of LOC behind a single public method doing complex calculations (eg proprietary lossy compression), and no externally meaningful intermediates produced by private functions getting good code coverage while only doing blackbox testing isn't practical. There aren't any mainstream automated testing frameworks not designed around the unit testing concept which leaves either using one to test private methods (offend the unit testing purists, or expose private methods just to test (and risk consumers code breaking because they tried to use methods they shouldn) – Dan Is Fiddling By Firelight Mar 28 '11 at 04:21
2

@JDPeckham I'm not saying that conventions of what should/shouldn't be done with a tool aren't important, but at the end of the day getting the job done is the most important thing. If that means pounding a nail with the back side of a hatchet because the tool vendors don't sell hammers then the hatchet manufacturers are just going to have to live with being outraged. – Dan Is Fiddling By Firelight Mar 28 '11 at 04:24
@ThomasEyde NUnit(Lite) tags are not supported by Visual Studio. In this case both NUnit(Lite) and MSUnit tests are shown in Test View window – hellboy Mar 11 '14 at 10:43
Scroll down to see the other way (NUnit->VS)... – Tuomas Hietanen Aug 26 '14 at 12:46
@DanNeely - the purists wouldn't get an electrician to check their house wiring. After all that is private and not consumer facing. Just turn on the mains and if anything goes bang that's a fail. Having said that getting code in a testable state is a good thing. One thing you can do is have internal classes and give just your test .dll access to them to do deeper testing. Then you can test just the public properties of those internal classes. – Martin Capodici Jan 20 '16 at 01:35

score 34 · Answer 3 · edited Aug 23 '20 at 19:26

34

Benefits/changes of the Visual Studio 2008 built-in unit testing framework:

The 2008 version now is available in professional editions (before it required expensive versions of Visual Studio, and this is just for developer unit testing) that left a lot of developers with the only choice of open/external testing frameworks.
Built-in API supported by a single company.
Use the same tools to to run and create tests (you may run them using the command line also MSTest).
Simple design (granted without a mock framework, but this is a great starting point for many programmers).
Long term support granted (I still remember what happened to NDoc, and I don't want to commit to a testing framework that might not be supported in five years, but I still consider NUnit a great framework).
If using Team Foundation Server as your backend, you can create work items or bugs with the failed test data in a simple fashion.

edited Aug 23 '20 at 19:26

Peter Mortensen

30,738
21
105
131

answered Nov 18 '08 at 16:49

Simara

1,135
1
12
14

4

I do think it's still telling about Microsoft's view of testing that it's NOT in the Standard version, just Professional and above. – J Wynia Mar 02 '09 at 15:27
Agree, I would love to see it in standard and up. In express versions it will be overkill for a beginner. – Simara Mar 05 '09 at 22:48
1

@J Wynia: Reading their decision to only include it in Professional and above as saying something about their view on testing is reading way too much into it. It's more likely a business decision than it is a philosophical one. – jason Nov 09 '10 at 02:39
@Simara Testing shoud be inherent to the development lifecycle. It shoud be provided in express editions. – eastender Sep 17 '13 at 06:00

score 33 · Answer 4 · edited Jul 30 '20 at 09:18

33

I have been using NUnit for two years. All is fine, but I have to say that the unit testing system in Visual Studio is pretty nice, because it's inside the GUI and can more easily do a test for private function without having to mess around.

Also, the unit testing of Visual Studio lets you do covering and other stuff that NUnit alone can't do.

edited Jul 30 '20 at 09:18

Peter Mortensen

30,738
21
105
131

answered Sep 18 '08 at 14:13

Patrick Desjardins

136,852
88
292
341

I wonder how could I test private function using VS? I have found only way changing it scope to public. Is there other method? – Alexander Prokofyev Oct 02 '08 at 13:03
Right click in the private function and select 'Create Private Accessor' – Robin Bennett Nov 27 '08 at 17:47
44

You shouldn't touch your privates. All kidding aside, one school of thought is that all you need the test are you'r public methods. Calling all your public methods should call all your private methods. If a private method is not being called through a public one, the private method is redundant. – Lieven Keersmaekers Mar 08 '09 at 10:30
7

@Lieven: if you are testing privates though the public you are really doing an intergration test and not a unit test. (Of course I am not a TDD zealot and I would probably just test the publics... but I feel like helping to start a fight) – Matthew Whited Jul 08 '09 at 20:52
11

@Matthew, integration testing is testing two or more units together. Testing private methods is just a (huge) violation of encapsulation and may lead to brittle unit tests that have to be modified everytime the implementation changes. – Omer Rauchwerger Aug 19 '09 at 10:15
+1 for mentioning code coverage in Visual Studio, which is hugely awesome, you can double-click a class or method and the editor highlights the code that is covered and the code that isn't covered by your unit tests. Makes it easy to get close to 100% code coverage. – J c Oct 12 '09 at 11:42
3

You can also use [assembly:InternalsVisibleTo(...)] with a compiler directive to take it out when you're making an RTM build. – Iain Galloway Nov 26 '09 at 10:03
1

I touch my privates all the time :) I think there is value in specifically testing private members to avoid the overhead from testing callers that require a lot of complex setup. – Crackerjack May 20 '10 at 20:15
The way most TDD-fanatics do it now, interface-methods have taken the place of public-methods, and public-methods have taken the place of private methods. So you have "public" methods, but can't access them through the interface, so really they are private-yet-testable. Many will try to justify this one way or another, but really they are lying to themselves - the only reason they do it this way is because you need an interface to mock, and cannot test private methods. (cont.) – BlueRaja - Danny Pflughoeft Nov 08 '10 at 22:46
(cont.) If the technology allowed them to test classes without creating an interface for *everything*, most of them would. And they would test their private-methods, as they should - you want to test the smallest possible units, to make the scope of any failures as small as possible. – BlueRaja - Danny Pflughoeft Nov 08 '10 at 22:47
THE GUI ISSUE: The magic Resharper (look it up) has a test runner for Visual Studio. THE PRIVATE ISSUE: If you want to test a private method, odds are that it shouldn't be private -- probably extracted to a separate class and injected instead. Being able to test a private method is a minus in my book. – Morten Apr 05 '11 at 13:39

score 14 · Answer 5 · edited Jul 30 '20 at 11:34

14

One slight annoyance of Visual Studio's testing framework is that it will create many test run files that tend to clutter your project directory - though this isn't that big of a deal.

Also, if you lack a plugin such as TestDriven.NET, you cannot debug your NUnit (or MbUnit, xUnit, etc.) unit tests within the Visual Studio environment, as you can with the Microsoft Visual Studio testing framework, which is built in.

edited Jul 30 '20 at 11:34

Peter Mortensen

30,738
21
105
131

answered Sep 19 '08 at 22:35

Tarsier

2,329
24
17

3

You can debug NUnit testing within Visual Studio 2005. – Jason Short Sep 24 '08 at 17:21
You can also debug xunit but it's non-obvious how to set that up (properties page) – annakata Jan 19 '09 at 10:40
1

You can easily debug NUnit by attaching the debugger to the running NUnit process like grimus said. No real disadvantage here. – Anne Schuessler Nov 19 '10 at 08:58
1: Number of tests configurable in VS settings - I set it to one. 2: Agree with above comments - possible but awkward, 3: Overall, I prefer the built in VS testing environment. – RaoulRubin Jul 05 '13 at 18:05

score 14 · Answer 6 · edited Aug 23 '20 at 19:29

14

Slightly off-topic, but if you go with NUnit I can recommend using ReSharper - it adds some buttons to the Visual Studio UI that make it a lot easier to run and debug tests from within the IDE.

This review is slightly out-of-date, but explains this in more detail:

Using ReSharper as an essential part of your TDD toolkit

edited Aug 23 '20 at 19:29

Peter Mortensen

30,738
21
105
131

answered Nov 28 '08 at 11:28

Richard Ev

52,939
59
191
278

With the Gallio plugin to R# you can run MSTest as well. – Kjetil Klaussen Feb 05 '09 at 12:00
CodeRush puts icons on the tests right in the code so you can run one test, or all the tests in a class or in a namespace. See here: http://community.devexpress.com/blogs/markmiller/archive/2009/11/16/the-test-runner-you-ve-been-waiting-for.aspx – Ryan Lundy Mar 26 '10 at 20:51
Resharper will run MSTest tests also. – JDPeckham Mar 28 '11 at 02:31

score 11 · Answer 7 · edited Aug 23 '20 at 19:55

I have done some TDD using both and (maybe I'm a little dumb) NUnit seems to be a lot faster and simpler to use to me. And when I say a lot, I mean a lot.

In MSTest, there is too many attributes, everywhere - the code that do the real tests is the tiny lines you may read here and there. A big mess. In NUnit, the code that do the test just dominates the attributes, as it should do.

Also, in NUnit, you just have to click on the tests you want to run (only one? All the tests covering a class? An assembly? The solution?). One click. And the window is clear and large. You get clear green and red lights. You really know what happens in one sight.

In VSTS, the test list is jammed in the bottom of the screen, and it's small and ugly. You have to look twice to know what happened. And you cannot run just one test (well, I did not find out yet!).

But I may be wrong, of course - I just read about 21 blog posts about "How to do simple TDD using VSTS". I should have read more; you are right.

For NUnit, I read one. And I was TDDing the same day. With fun.

By the way, I usually love Microsoft products. Visual Studio is really the best tool a developer can buy - but TDD and Work Item management in Visual Studio Team System sucks, really.

score 11 · Answer 8 · edited Aug 23 '20 at 19:45

My main beef with Visual Studio unit tests over NUnit is the Visual Studio test creation tends to inject a bunch of generated code for private member access.

Some might want to test their private methods, and some may not. That's a different topic.

My concern is when I'm writing unit tests they should be extremely controlled so I know exactly what I'm testing and exactly how I'm testing it. If there's auto generated code I'm losing some of that ownership.

score 11 · Answer 9 · edited Jul 30 '20 at 11:27

11

xUnit is another possibility for a greenfield project. It's got perhaps a more intuitive syntax, but it is not really compatible with the other frameworks.

edited Jul 30 '20 at 11:27

Peter Mortensen

30,738
21
105
131

answered Sep 18 '08 at 19:46

Ben Fulton

3,988
3
19
35

score 9 · Answer 10 · edited Jul 30 '20 at 11:30

First I want to correct a wrong statement: you can run MSTest outside of Visual Studio using the command line. Although several CI tools, such as TeamCity, have better support for NUnit (probably would change as MSTest becomes more popular).

In my current project we use both and the only big difference we found that MSTest always runs as a 32 bit while NUnit runs as either 32 bit or 64 bit tests which only matters if your code uses native code that is 32/64 bit dependent.

score 9 · Answer 11 · edited Aug 23 '20 at 19:43

9

I got messages that "NUnit file structure is richer than VSTest"... Of course, if you prefer the NUnit file structure, you can use this solution to the other way, like this (NUnit → Visual Studio):

 #if !MSTEST
     using NUnit.Framework;
 #else
     using Microsoft.VisualStudio.TestTools.UnitTesting;
     using TestFixture = Microsoft.VisualStudio.TestTools.UnitTesting.TestClassAttribute;
     using Test = Microsoft.VisualStudio.TestTools.UnitTesting.TestMethodAttribute;
     using SetUp = Microsoft.VisualStudio.TestTools.UnitTesting.TestInitializeAttribute;
     using TearDown = Microsoft.VisualStudio.TestTools.UnitTesting.TestCleanupAttribute;
 #endif

Or any other conversion... :-) This use here is just an alias to the compiler.

edited Aug 23 '20 at 19:43

Peter Mortensen

30,738
21
105
131

answered Feb 13 '09 at 18:53

Tuomas Hietanen

4,650
2
35
43

1

I don't understand what you are saying here. – PositiveGuy Feb 11 '10 at 19:30
no fixture level setup/teardown :( or do they assume we use ctor and dtor? – JDPeckham Mar 28 '11 at 02:33

score 8 · Answer 12 · edited Aug 23 '20 at 19:52

I started with MSTest, but I switched for one simple reason. MSTest does not support inheritance of test methods from other assemblies.

I hated the idea of writing the same test multiple times. Especially on a large project where test methods can easily run into 100's of tests.

NUnit does exactly what I need. The only thing that is missing with NUnit is a Visual Studio addin which has can display the red/green status (like VSTS) of each test.

score 7 · Answer 13 · edited Aug 23 '20 at 20:00

If you are considering either MSTest or NUnit, then I recommend you look at MbUnit. My reasons are

TestDriven.Net compatibility. Nothing beats have TestDriven.Net.ReRunWithDebugger bound to a keyboard combination.
The Gallio framework. Gallio is a test runner like NUnit's. The only difference is it doesn't care if you wrote your tests in NUnit, MSTest, xUnit or MbUnit. They all get run.
Compatibility with NUnit. All features in NUnit are supported by MbUnit. I think you don't even need to change your attributes (will have to check that), just your reference and usings.
Collection asserts. MbUnit has more Assert cases, including the CollectionAssert class. Basically you no longer need to write your own tests to see if two collections are the same.
Combinatorial tests. Wouldn't it be cool if you could supply two sets of data and get a test for all the combinations of data? It is in MbUnit.

I originally picked up MbUnit because of its [RowTest ....] functionality, and I haven't found a single reason to go back. I moved all my active test suites over from NUnit and never looked back. Since then I've converted two different development teams over to the benefits.

score 6 · Answer 14 · edited Jul 30 '20 at 11:33

As far as I know, there are four frameworks available for unit testing with .NET these days:

NUnit has always been out in front, but the gap has closed in the last year or so. I still prefer NUnit myself, especially as they added a fluent interface a while back which makes tests very readable.

If you're just getting started with unit testing it probably doesn't make much difference. Once you're up to speed, you'll be in a better position to judge which framework is best for your needs.

score 6 · Answer 15 · edited Aug 23 '20 at 20:02

6

I don't like the Visual Studio's built-in testing framework, because it forces you to create a separate project as opposed to having your tests as part of the project you're testing.

edited Aug 23 '20 at 20:02

Peter Mortensen

30,738
21
105
131

answered Apr 21 '10 at 14:17

Gene Mitelman

69
1
1

3

You can actually fool Visual Studio by manually editing the project files and adding in the ProjectTypeGuids values that it uses as to identify test projects: <ProjectTypeGuids>{3AC096D0-A1C2-E12C-1390-A8335801FDAB};{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}</ProjectTypeGuids> – Paul Ruane Aug 03 '10 at 16:09

score 5 · Answer 16 · edited Aug 23 '20 at 19:36

5

MSTest is essentially NUnit slightly reworked, with a few new features (such as assembly setup and teardown, not just fixture and test level), and missing some of the best bits (such as the new 2.4 constraint syntax). NUnit is more mature, and there is more support for it from other vendors; and of course since it's always been free (whereas MSTest only made it into the Professional version of Visual Studio 2008, and before that it was in way more expensive SKUs), and most ALT.NET projects use it.

Having said that, there are some companies who are incredibly reluctant to use something which does not have the Microsoft label on it, and especially so OSS code. So having an official Microsoft test framework may be the motivation that those companies need to get testing; and let's be honest, it's the testing that matters, not what tool you use (and using Tuomas Hietanen's code, you can almost make your test framework interchangeable).

edited Aug 23 '20 at 19:36

Peter Mortensen

30,738
21
105
131

answered Jan 14 '09 at 03:23

David Keaveny

3,904
2
38
52

I love NUnit's contraint syntax but I think you should read this regarding the `SetUp` and `TearDown` attributes: http://jamesnewkirk.typepad.com/posts/2007/09/why-you-should-.html – Nobody Dec 02 '10 at 16:34
What is *"SKU"*? I suppose it is not [Young Communist League of Sweden](https://en.wikipedia.org/wiki/Young_Communist_League_of_Sweden) (*Sveriges Kommunistiska Ungdomsförbund*). – Peter Mortensen Aug 23 '20 at 19:39
@PeterMortensen it's a Stock Keeping Unit, apparently, for tracking inventory - inventory of software products in this case. So each version of Visual Studio has its own SKU - one for Professional, one for Enterprise. – David Keaveny Sep 21 '20 at 00:24
But maybe young Swedish communists like Microsoft products more than most people? – David Keaveny Sep 21 '20 at 00:25

score 4 · Answer 17 · answered Nov 09 '10 at 02:35

With the release in .NET 4.0 of the Code Contracts system and the availability of a static checker, you would need to theoretically write fewer test cases and a tool like Pex will help identify those cases. Relating this to the discussion at hand, if you need to do less with your unit tests because your contracts are covering your tail, then why not just go ahead and use the built-in pieces since that is one less dependency to manage. These days, I am all about simplicity. :-)

NUnit vs. Visual Studio 2008's test projects for unit testing

18 Answers18

Linked

Related