Is there anything I can do in NUnit that I can't do in MSTest?

Question

This question has been asked in various forms in a number of different forums, but, IMHO, I haven't been able to find a place where it's really answered clearly, so I'm going to reframe it and ask it again.

I work in a basically Microsoft Shop. We use TFS, and all of our developers have MSDN subscriptions including the Team Suite edition of VS. So we have access to MSTest.

I've read the various NUnit vs. MSTest comparisons, and the developer community seems to pretty much overwhelmingly choose NUnit. But the reasons given don't ever seem to be overwhelming or compelling, at least to our situation. (NUnit is updated more often, NUnit is faster, NUnit doesn't require TFS, etc.)

I can use NUnit if I choose, but the use of open source software without a formal support behind it has to be defended. I need a fairly compelling reason to do so.

What I basically have to answer to justify using NUnit in preference to MSTest is this: is there anything that I can do in NUnit that I can't do with comparable effort in MSTest?

Answer 1

• NUnit contains a [TestCase] attribute that allows implementing parametrized tests. This does not exist out of the box in MSTest - it can be done via extensibility though.
• MsTest's ExpectedException attribute has a bug where the expected message is never really asserted even if it's wrong - the test will pass.
• NUnit ships with an Assert.Throws API to allow testing an exception on a specific line of code instead of the whole method. A similar feature exists for MSTest (implemented by the same person who did it for NUnit) but does not ship with MSTest.
• NUnit contains a fluent version of Assert API out of the box. MSTest has third party extensions that do this, but none are shipped with MSTest.
• NUnit allows abstract classes to be test fixtures (so you can inherit test fixtures). MsTest allows this but limits abstract classes to a single assembly.
• NUnit allows non public classes to be test fixtures (as of the latest version)
• NUnit was created SOLELY for the idea of unit testing. MSTest was created for Testing - and also a bit of unit testing.
• NUnit contains PNunit (running parallel tests with NUnit). MSTest added this ability in Visual Studio 2010 which is configurable via XML

Answer 2

Roy, Bunch of your information is out of date especially as it relates to 2010;

Nunit contains a [TestCase] attribute that allows implementing parametrized tests. this does not exist in MSTest

This can be implemented using unit test extensibility in 2010.

MSTest's ExpectedException attribute has a bug where the expected message is never really asserted even if it's wrong - the test will pass.

Correct thats still there

NUnit has an Assert.Throws API to allow testing an exception on a specific line of code instead of the whole method (you can easily implement this one yourself though)

Jim implented a version of Assert.Throws for MSTest at the same time as he did the original implementation for NUnit, NUnit has included in subsequent releases, MSTest has not, its still possible to use though.

NUnit contains a fluent version of Assert API (as already mentioned - Assert.That..)

There are several of these implemented by 3rd parties for MSTest

NUnit is much faster

See Jamie's comment he has managed to get MSTest running faster :-)

NUnit can run tests in 32 and 64 bit (MSTest only runs them in 32 bit IIRC)

Not in 2010, 64 bit support is built in.

NUnit allows abstract classes to be test fixtures (so you can inherit test fixtures). MsTest does not.

This works but not across assemblies which does limit its usefulness.

NUnit allows non public classes to be test fixtures (as of the latest version)

Still there

NUnit was created SOLELY for the idea of unit testing. MSTest was created for Testing - and also a bit of unit testing.

Correct, there is a lot of misconception that MSTest is the same as Nunit but MSTest is a generalized framework.

NUnit contains PNunit (running parallel tests with NUnit). MSTest only adds this ability in vs 2010

Correct there is an XML config setting that allows control of degree of parallelism.

Answer 3

NUnit has a richer assert API. The api is particularly elegant (fluent, even), for example

Assert.That(Is.Unique, myResults);  // assert: myResults is a collection of unique items

If you've seen the Hamcrest extensions to JUnit you'll recognise this style.

It also has a growing set of extensions, such as performance testing and an excellent VS plugin.

Answer 4

I can point you to a couple of blogs on frustrations with MSTest:

• MSBuild, NAnt, NUnit, MSTest, and frustration
• Clean build server with MSTest == FAIL

To be fair, these people are trying to set up MSTest on non-TFS build servers. Some of their issues are not going to apply to your situation.

We are primarily a Microsoft shop, and use TFS for source control. However, we use TeamCity for Continuous Integration; we like it, and it integrates reasonably well with TFS. I've never used MSTest; we've been using NUnit for years, and have seen no reason to change.

MSTest is supposed to have tight integration with Team Suite, which (since your company has already paid the outrageous fee for that) is a point in its favor.

NUnit comes with less vendor lock-in, and has a rich API. As serg10 pointed out, the Assert.That syntax is particularly powerful and elegant.

In the end, you can write good unit tests without all the fancy features. Some of them may even get in the way (which is the theory behind xUnit.net). I would recommend that your team standardize on one test framework; avoid having some code in MSTest and other code in NUnit.

I think writing good tests is more important than your choice of frameworks. Consider reading The Art of Unit Testing: with Examples in .NET, writing some tests, then seeing if MSTest is adequate for your team's needs.

EDIT: Appendix B of The Art of Unit Testing has some good comments on Microsoft's Unit Testing Framework. It mentions YUnit as an example of how cumbersome it is to extend MSTest. However, the author does suggest MSTest for Team System users due to the tight integration.

Answer 5

I have a nice way in between MsTest and NUnit. You can use the MSTest framework to run your test (TestClass attribute and TestMethod attribute for each test) but use the NUnit Assert API.

just do this :

using Microsoft.VisualStudio.TestTools.UnitTesting; 
using Assert = NUnit.Framework.Assert;  

now you can use Assert.That(..) and still have TFS automated build and test report. your test can be run in VS, TestDriven.net or Resharper, it doesnt matter, the test will fail or pass correctly, and the fail output will be according to NUnit framework.

see here : http://alsagile.com/archive/2010/03/09/stop-the-war-between-nunit-and-mstest-make-them.aspx

Answer 6

Please read ExpectedExceptionAttribute documentation carefully, it is nowhere stated that tests the exception message so it's not a bug that is not asserted.

The second parameter is the assert message what is displayed when expected exception type is not thrown, not the expected exception message. I.e. like the second parameter in Assert.IsTrue(condition, message).

Answer 7

1. The MSTest test runner is non-deterministic, which should be enough to scare you away from using it. I cannot consistently run MSTest from TFS 2010 using the integrated test runner; it breaks with a few different error messages (and, inconsistently) across project builds and across build agents.
2. MSTest sometimes (not consistently) breaks because it leaks memory--out of memory exceptions still happen for us, even with the newest version of Visual Studio. The workaround for this problem is horrible.
3. I have other minor quibbles with MSTest, and have blogged a bunch of workarounds for using MSTest in general: http://www.pseale.com/blog/TFSAsYourBuildCIServerOnlyPositiveTakeaways1Of2.aspx

I found this question as I am switching us to NUnit from MSTest because we can't trust the result of our CI build anymore because of MSTest. Switching to NUnit will allow us to (finally) trust that a CI build failure is real, not another MSTest glitch. This is the real answer--only with NUnit (or some other non-MSTest test runner) can we trust our CI build.

Answer 8

@Dave Hanna MS Test is available out of the box so one less component to worry about deploying and keeping updated. It is also integrated with Visual Studio and other Microsoft products by design.

The fluent syntax is nice but it's not a functional difference so I'd ignore it. Not to mention that you can have the same in MS Test by using an extension library.

Performance. 99% of test performance is controlled by system under test, so even a 100% performance difference between two library would result in an ignorable overall difference in performance.

Open source vs. not open source: the chances that you'll benefit from using an open source library are minimal. The exceptions are when changes are pulled into trunk frequently or when there are enough resources to keep the modified branch updated.

Answer 9

I've been working on first class support for MSTest in TestDriven.Net 3.0. In previous versions the MSTest support was very basic (there seemed to be few people using MSTest).

As of TestDriven.Net 3.0 Beta 2, there is pretty comprehensive support for all of the unit testing related attributes of MSTest. There is even support for data driven tests using the DataSource attribute. Your tests will also execute at close to NUnit speeds!

If you use MSTest, I'd be interested to hear if any of your unit tests fail when executed using TestDriven.Net 3.0 Beta 2 (or later).

Please try it on your MSTest projects and let me know how you get on: http://www.testdriven.net/download.aspx

NOTE, it doesn't support the DeploymentItem or HostType attributes. You can use the 'Copy to Output Directory' project item setting instead of DeploymentItem.

Answer 10

You can change the code in NUnit because it is open source. You can't do that with MSTest.

Answer 11

See http://fluentassertions.codeplex.com. You can do stuff like

"ABCDEFGHI".Should().StartWith("AB").And.EndWith("HI").And.Contain("EF").And.HaveLength(9);

new[] { 1, 2, 3 }.Should().HaveCount(4, "because we thought we put three items in the 
collection"))

dtoCollection.Should().Contain(dto => dto.Id != null);

collection.Should().HaveCount(c => c >= 3);

dto.ShouldHave().AllPropertiesBut(d => d.Id).EqualTo(customer);

dt1.Should().BeWithin(TimeSpan.FromHours(50)).Before(dt2); 

Action action = () => recipe.AddIngredient("Milk", 100, Unit.Spoon);
action
   .ShouldThrow<RuleViolationException>()
   .WithMessage("Cannot change the unit of an existing ingredient")
   .And.Violations.Should().Contain(BusinessRule.CannotChangeIngredientQuanity

Answer 12

You can very easily run NUnit tests on Linux with Mono and I don't think you can do that with tests for MSTest.