119

I'm using Visual studio (sometimes resharper) to run my unit test.

I heard about NUnit, but I don't know many things about it...

Should I care about it ? Can it offer something better than visual studio?

Should I Use NUnit and why?

Guilherme Fidelis
  • 1,022
  • 11
  • 20
Tim
  • 2,887
  • 10
  • 33
  • 33

9 Answers9

104

NUnit has few advantages over MS-Test

  1. Suite attribute - can aggregate tests and execute them separately (useful for large projects with fast and slow tests for example)
  2. Readable Assert method, e.g. Assert.AreEqual(expected, actual) vs Assert.That(actual, Is.EqualTo(expected))
  3. NUnit has frequent version updates - MS-Test has only one per VS version.
  4. Many integrated runners including Resharper and TestDriven.NET
  5. Expected exception message assertion - can be done using attribute in NUnit but must be done using Try-Catch in MS-Test
  6. [TestCase]! NUnit allows for parameter-ized tests.
Jess
  • 23,901
  • 21
  • 124
  • 145
Elisha
  • 23,310
  • 6
  • 60
  • 75
  • 32
    Exception can also be asserted by attribute in MS-Test: ExpectedExceptionAttribute. – Stefan Steinegger Oct 12 '09 at 12:11
  • 9
    I would use NUnit with Assert.Throws<>() because this follows the AAA-Principle, which is not through for the attribute way. – Oliver Hanappi Oct 12 '09 at 12:26
  • @Stefan Steinegger, MSTest has expected exception attribute but it cannot verify the message text – Elisha Oct 16 '09 at 14:27
  • 2
    NUnit has a standalone runner thats easy and light to install – Ruben Bartelink Dec 17 '09 at 09:49
  • 25
    #3 is not a feature, it's a problem and #5 is 100% false; MS Test has the ExpectedException attribute, *and always has*. – Randolpho Dec 17 '09 at 15:14
  • 2
    @Randolpho, ExpectedException exists, but it doesn't support message assertion. The missing feature is only the thrown exception message assertion, not the thrown exception itself. I can assert using MS-Test attribute if the exception was thrown but if I want to verify it contains the right message it must be done using try-catch. – Elisha Dec 17 '09 at 15:25
  • 4
    @Elisha As I recall, they deliberately did not support message text checking because the text is not seen to be significant and can be arbitrary. That is, it doesn't affect the flow of a program. – Rob Kent Sep 20 '11 at 11:57
  • 7
    "#3 is not a feature, it's a problem" - doesn't sound like a very agile way of thinking... – SamuelKDavis Jan 04 '15 at 23:31
  • 2
    Which of those expressions in point 2 is NUnit and which is MSTest? – kdbanman Jul 20 '15 at 22:29
  • 1
    Assert.That(actual, Is.EqualTo(expected)) is Nunit, the other one is MSTest. – Elisha Jul 24 '15 at 08:18
73

From my current perspective (after 8 months of development with about 10 developers on average) I would advise against using MSTest for the following reasons

  • The framework in itself is quite slow. I don't mean the test code that you write - that's under your control. I mean the framework running those tests is slow, whether it's running a test suite, single tests etc.
  • The need to keep a Test-Metadata file which always leads to complications when several developers are working on it (recreating e.g. the metadata etc.). Every other test suite doesn't need a metadata file. It is kind of nice to organize your tests but you can achieve the same through namespaces, classes and method names.
  • Doing Continuous Integration, if you want to run unit tests on your build machine you will need to install Visual Studio on that machine.

In other words, if I would have to decide again 8 months ago, I would probably take NUnit. I may not have the integrated test results report, but developers would have a more seamless testing experience.

kdbanman
  • 10,161
  • 10
  • 46
  • 78
flq
  • 22,247
  • 8
  • 55
  • 77
  • 6
    +1, avoid MSTest unless you don't have a choice. The various open-source frameworks are better (xUnit, NUnit, MbUnit, etc). – Brannon Dec 17 '09 at 09:54
50

Here is my experience with MS Test

  • We are running MS Test with around 3800 Test.
  • It takes very long for the tests just to start executing, which is painful when running single tests.
  • It takes around 1GB Memory to execute the tests. No, it is not due to memory leaks in our tests. Frequently we run into OutOfMemoryExceptions.
  • Because it uses that much resource, we are starting to execute the tests from batch-files. So what's the whole integration good for?
  • It is buggy and unstable:
    • For instance, if you remove the [Ignore] Attribute from a test, it does not recognize it, because it caches information about tests somewhere. You need to refresh the testlist, which sometimes solves the problem, or restart VS.
    • It randomly does not copy reference assemblies to theout directory.
    • Deployment Items (additional files to be used) just don't work properly. They are ignored randomly.
  • There is hidden (not visible in the test code) information in vsmdi and testrunconfig files. If you don't care about it, it might not work.
  • Functionally it might be comparable to NUnit, but it is very expensive if you consider using VS tester edition.

Addition: We have some more tests now, can't even say how many. It is impossible to run them all anymore from Visual Studio, because of OutOfMemoryExceptions and other instability problems. We run the tests from scripts. It would be easy to view test results in Visual Studio, but when the solution is open, VS crashes (every time). So we need to search the failing tests using text search. There is no advantage of an integrated tool anymore.

Another Update: We are using VS 2013 now. A lot of things changed. They rewrote the MS Test test runner for the third time since we started. This caused a lot of breaking changes, but neither new version was doing anything better. We are glad that we didn't use the fancy features of MS Test, because they are all not supported anymore. It's really a shame. We are still using scripts to build and run all unit tests, because it is handier. Visual Studio required a few minutes to start running tests (time measures after compilation until first test starts). They probably fixes it with an update and this might be a specific problem of our project. However, Resharper is much quicker when running the same tests.

Conclusion: At least in combination with Resharper, MS Test is useful. And I hope that they finally find out how the test runner should be written and won't do this kind of breaking changes when we update Visual Studio next time.

Stefan Steinegger
  • 63,782
  • 15
  • 129
  • 193
  • I've recently started running them without debugging, makes it *much* faster and much more like how NUnit might be used, but it's still sucky. It seems that the poor performance is related to Visual Studio doing funky stuff when debugging. (That is, use "ctrl+F5" rather than just "F5" - you still get the nice "integration" with VS) – Arafangion Dec 13 '10 at 08:06
18

NUnit can be used in combination with visual studio. It is a framework not a separate program. So you could care an see if it suits you :).

alt text
(source: codeplex.com)

"After installing the plugin you'll find a new submenu under the tools menu."

See http://nunitit.codeplex.com/ for more information on importing it.

Also, a lot can be found using the search of SO. This topic lists advantages of NUnit over MS standard testing for instance.

Glorfindel
  • 21,988
  • 13
  • 81
  • 109
bastijn
  • 5,841
  • 5
  • 27
  • 43
6

Biggest advantage MS-Test over NUnit is MS-Test can generate mock objects using Reflection. I found it very usefull

Maciej
  • 10,423
  • 17
  • 64
  • 97
  • 12
    http://code.google.com/p/moq/ , http://www.nmock.org/ , http://stackoverflow.com/questions/37359/what-c-mocking-framework-to-use – David Schmitt Feb 17 '10 at 09:41
3

NUnit works with the Standard edition of VS.

Per Erik Stendahl
  • 863
  • 6
  • 14
1

NUnit is a unit testing framework, which is also supported by resharper. I think you are using Microsoft's unit testing framework, so NUnit is just an alternative to Microsoft's product ;)

Here is the link to the homepage of NUnit: http://nunit.org

Lucas
  • 1,149
  • 1
  • 9
  • 23
Oliver Hanappi
  • 12,046
  • 7
  • 51
  • 68
  • so you mean that nunit doesn't bring anything else than microsoft unit testing framework ? – Tim Oct 12 '09 at 11:30
  • It does, see my link for instance in my post (above). – bastijn Oct 12 '09 at 11:34
  • I'm using NUnit and don't really know about Microsoft's unit testing framework, so I can't say what's better. I think there is a good chance that you will find some topic on the differences here on SO. – Oliver Hanappi Oct 12 '09 at 11:35
  • It provides some useful constructs such as [TestCase] for running one test method with different args, [Theory] - for building more elaborate specifications and others. It also supports very nice fluent syntax for assertions. And, last but not the least, it is used much wider than MSTest, so you'd have better chance to get support/info if you'd get into trouble. – elder_george Oct 12 '09 at 11:37
1

I am not sure of others but NUnit provides nice GUI and console to run your Unit Tests and also you can generate report of the result of NUnit Test execution which would give the detail of whethere the test has failed or passed and also what time did it take for your unit test

Vinay Pandey
  • 8,589
  • 9
  • 36
  • 54
0

In NUnit, tests are not executed in parallel. Rather, it appears that all tests execute on a single thread. In MSTest, each test is instantiated on a separate thread, this results the runs being interleaved. Therefore, if test A depends on test B for its success, it likely will fail as test B will likely start running as test A is running.

Mourad Samy
  • 9
  • 1
  • 3