9

QUESTION - If I'm starting a Windows Forms application in C# using Visual Studio 2008, which unit testing framework should I use?

There seems to be one build into VS2008? Or should I look into something like NUnit? Or is NUnit what is used in VS2008 under the bonnet? What's the most popular.

Would be good to have something that allows for: (a) mocking/stubs, and (b) ability to assert exceptions

thanks

Greg
  • 34,042
  • 79
  • 253
  • 454

6 Answers6

8

I would recommend using NUnit for your testing framework. It's very lightweight and easy to ship around if you need to get it set up on a build server. This isn't the case using MSTest. As for mocking/isolation frameworks, Rhino Mocks has the largest user base and you're likely to find answers to your questions quickest with Rhino Mocks.

Chris Missal
  • 5,987
  • 3
  • 28
  • 46
  • 1
    The biggest disadvantage with MSTest is once you start down that road it's hard to change your mind. NUnit is a little easier to convert to MSTest. DO try them both. – csharptest.net Oct 08 '09 at 02:07
5

If it just a simple application, the MSTest framework built into VS2008 should tick all the boxes. It is not the same as NUnit, though a lot of people (mistakenly) refer to it as if they were the same thing.

I have never used it for mocking, so maybe someone else can elaborate on that. It generates a separate project in your solution that is just for the purpose of testing. This is a good explanation of how to use it.

alt text
(source: geekzone.co.nz)

Edit: Doing a bit more digging, I came across this comparison of MSTest and NUnit.

Drawbacks of NUnit Framework:

  • Installation of NUnit comes in a separate MSI.
  • No Integration with Visual Studio.
  • Requires writing test cases manually.
  • No auto generation of code.
  • Requires opening separate window (NUnit Console or GUI) to execute test cases.
  • Ordering of test cases execution not available.
  • No inbuilt feature to debug test cases.
  • No inbuilt feature to enable/disable test cases.
  • No inbuilt feature to give additional information of test cases like stack trace, Trace Information etc.
  • No inbuilt feature to sort test cases based on Computer Name, Class Name and Host Type etc.

That's from the comparison article I linked to.

I have never used NUnit, only MSTest for C# and JUnit for Java, so I am kind of biased in that respect. MSTest has always worked really well for me for WinForms, with features like being able to run the tests individually, show really detailed reports (with individual trace logs) and autogenerate all the boilerplate test code and do all those kind of things that make VS2008 such a brilliant IDE. By the sounds of it, NUnit has worked well for others and they have their reasons why they like it.

Unless you have a particular reason to take the NUnit/Testdriven.NET approach, like you need a certain feature, or you just prefer that way of setting up tests and trying to integrate it back into VS, then I don't see any reason not to just use MSTest which works right out of the box.

Glorfindel
  • 21,988
  • 13
  • 81
  • 109
Dale
  • 12,884
  • 8
  • 53
  • 83
  • 1
    Seems unfair when a lot of people use TestDriven.NET to drive NUnit. That provides great integration and removes most of the drawbacks you mention. The NUnit [Ignore] attribute disables [Test] methods or [TestFixture] classes. Also, IMHO ordering test cases is a bit smelly :) – si618 Oct 08 '09 at 03:08
  • 4
    ouch. 600 chars is not going to be enough. 1. that's good. I can run my tests on any (build/QA) machine without having to install VSTS. 2. If you want right-click and run tests, get Resharper/TestDriven.Net add-ins. Also see SO q#196740. #3,4,6,10 - doesn't make sense. I believe programmers know their code better than VS. It follows that they can better identify what tests to write. But then I'm test-infected (read TDD zealot). test cases should not be order dependent - so not sure why that is a disadvantage. – Gishu Oct 08 '09 at 03:20
  • 2
    Contd... Debugging test cases can be done (add nunit as external exe to debug for your test dll) ; although that it implies that you don't know your code anymore. Sorting test cases - again not sure why you need it. But you could group your tests into categories like [Category("Slow Tests")] and then include or exclude them from test runs. You also have a checkbox view where you can manually tick tests you'd like to run. Finally 9. stacktrace - nunit does that in the first tab. A unit testing framework would be kinda crippled if it didnt point to the exact location where the test failed! – Gishu Oct 08 '09 at 03:23
  • 2
    "Installation of NUnit comes in a separate MSI." You can just reference the assembly from a project. This is how I always use it. – Chris Missal Oct 08 '09 at 05:05
  • 1
    -1 since so many "facts" are simply wrong (as already pointed out in other comments). – The Chairman Oct 08 '09 at 07:56
  • 5
    I am also a TDD person, I see this answer has been voted down a number of times. I didn't mean to upset the one-eyed NUnit fanatics, just explain what I used and why it works for me especially for Winforms apps. The comparison comes from the site I linked to. Apart from Gishu who says NUnit can indeed print a stacktrace when a test fails (that does seem kinda critical) the rest are just opinions or justifications for why NUnit does things differently. – Dale Oct 08 '09 at 18:45
  • This description seems incredibly biased. Having to write your testcase is not a drawback ! That seems just as logical as saying having to write code to write code is a downside – roundcrisis Feb 10 '12 at 15:16
5

We've started with MSTest. It has an advantage that is (mildly) difficult to replicate in NUnit. It allows you to access private members of the class. This is invaluable when you are checking state. For instance, say that I have a function that puts entries into a dictionary that is not exposed. My understanding, is that with NUnit you have to use reflection to get to the dictionary, or add a getter for the sole purpose of testing. Here, you can just check the dictionary. Very easy and very clean. It also allows you to test private functions, which I love (I know some people don't believe in this). While I don't love MSTest, that feature makes testing much easier.

VS integration is also nice.

Steve
  • 11,763
  • 15
  • 70
  • 103
  • how does MSTest achieve the "access private members" Steve do you know? just interested in why it would be easy for MSTest guys to provide this but not the NUnit guys – Greg Oct 08 '09 at 02:07
  • 1
    my understanding is the nunit guys don't believe in it. With MSTest, you just right click in the class you want to test and choose "Add Class Accessor". You then access class Foo as Foo_Accessor in your test. – Steve Oct 08 '09 at 02:11
  • My group is relatively new to unit testing, and there was a lot of things we liked about NUnit, but the private accessor stuff was a deal breaker. – Steve Oct 08 '09 at 02:13
  • 1
    It's also an encapsulation breaker.. You're walking right into an anti-pattern - Visit http://stackoverflow.com/questions/333682/tdd-anti-patterns-catalogue and look for inspector. NUnit is doing the right thing by not supporting this. – Gishu Oct 08 '09 at 02:36
  • 2
    We can agree to disagree on this one. My classes hold stuff and have state. I need to find a way to check that state without calling other functions (if I can help it). I've written the "perfect" classes where every function is self contained and has little/no state, but I haven't been able to do it for classes that actually do stuff. – Steve Oct 08 '09 at 02:47
  • Not trying to be dogmatic. If you haven't already read this take a look - http://www.pragprog.com/articles/tell-dont-ask . Peeking inside a class to look at its state - makes tests fragile. Your test now has implementation level knowledge.. tests should state what (and not how). The risk here is that future refactorings e.g. changing the type of variable holding the state or its name - break tests. Hence the term brittle or fragile tests. The behavior is not broken but the tests would still fail. – Gishu Oct 08 '09 at 03:12
3

Use MSTest. It's built into VS2008 and has support in the IDE for creating test functions. If and when you "outgrow" MSTest, then you should look at NUnit or the more modern xUnit. You can expect exceptions in MSTest by decorating with the ExpectedException attribute.

[Test]
[ExpectedException(typeof(ArgumentNullException))]
public void Should_throw_ArgumentNullException_when_the_Order_is_null()
{
    OrderProcessor processor = new OrderProcessor();
    processor.ProcessOrder(null);
}

Mocking frameworks are independent of your choice of testing framework. Popular choices for mocking framework are: Moq and Rhino Mocks.

Phillip Ngan
  • 15,482
  • 8
  • 63
  • 79
  • 1
    For sure. And you can also run NUnit from within Visual Studio if you have resharper. – Phillip Ngan Oct 08 '09 at 05:16
  • MSTest doesn't use the [Test] attribute. This is used in NUnit and MbUnit, which makes those two interchangeable, also a bonus for not using MSTest, you're more strongly tied into a framework. – Chris Missal Oct 08 '09 at 23:18
2

i would say, for simplicity, start with the one built into visual studio 2008. NUnit is fantastic and i use it alot, so you can probably move on to NUnit once your comfortable with writing unit tests.

Andrew Keith
  • 7,515
  • 1
  • 25
  • 41
  • does VS2008 unit tests allow you to assert that an exception should occur? can't quite even see this? – Greg Oct 08 '09 at 02:02
  • 2
    Yes, although you do it with an attribute ([ExpectedAssertion()] I believe) – Steve Oct 08 '09 at 02:12
1

You didn't mention if you were going to be test-driving code..

  • Go with NUnit if you're not familiar with any of the xUnit frameworks. Smallest learning curve. It has been around for longer. Charlie Poole is active on NUnit (open source) and has a good history of good updates.
  • VS2008 has MSTest integrated inside it. It's not the same as NUnit. I've never had reason to switch from NUnit. I'd argue that NUnit definitely has the biggest user-base.
  • NUnit has something bundled for Mocks but I'd definitely point you to Moq. Rhino Mocks has been the top dog for a while now but Moq has an easier learning curve (from online docs) although it may not always handle edge scenarios... yet IMHO
  • Asserting exceptions is supported in both.

I'd recommend getting a thin book 'Pragmatic Unit Testing in C# with NUnit' and going through it atleast once.

On the NUnit v MSTest line, I'm not fit to comment. 0 flying-time with MSTest.

  • Although I hear that VS2010 will offer trinkets like automated code-coverage reports.. only if you've MSTest tests. But that's the kind of argument I keep hearing in MSTests favor.. more on the lines of IDE Support and integration. There are great add-ins like Resharper and TestDriven that can patch that in but at a price.
  • Also IMHO, Microsoft's unit testing tooling has a subtle way of leading you down the dark path. so be careful. See my comment to Dale on this thread.
Gishu
  • 134,492
  • 47
  • 225
  • 308