We are running nightly builds which at the end run all of our UnitTest using the MsTest framework.
We must have 100% pass rate, so if one fails there is no point running the others; hence we will like to stop the execution on the first failed test.
Is there anyway we can do achieve it?
Are you really sure you want to throw away test results?
Say somebody has a bad day and introduces multiple bugs A, B, and C into your code. You might only find out about A on Monday, so you fix that, and you don't know about issue B until Tuesday, and then you don't fix C until Wednesday. But since you were missing test coverage for half a week, bugs introduced Monday aren't discovered until days after they were introduced, and it's more expensive and takes longer to fix them.
If it doesn't cost you much to keep running the tests after a failure, isn't that information useful?
That said, hacking together a fix in your test libraries wouldn't be that hard. Have all possible codepaths of test failure set a static variable StopAssert.Failed = true; (probably by wrapping calls to Assert and catching AssertFailedExceptions. Then just add [TestInitialize()] methods to each test class to stop each test after a failure!
public class StopAssert
{
public static bool Failed { get; private set; }
public static void AreEqual<T>(T expected, T actual)
{
try
{
Assert.AreEqual(expected, actual);
}
catch
{
Failed = true;
throw;
}
}
// ...skipping lots of methods. I don't think inheritance can make this easy, but reflection might?
public static void IsFalse(bool condition)
{
try
{
Assert.IsFalse(condition);
}
catch
{
Failed = true;
throw;
}
}
}
[TestClass]
public class UnitTest1
{
[TestInitialize()]
public void Initialize()
{
StopAssert.IsFalse(StopAssert.Failed);
}
[TestMethod]
public void TestMethodPasses()
{
StopAssert.AreEqual(2, 1 + 1);
}
[TestMethod]
public void TestMethodFails()
{
StopAssert.AreEqual(0, 1 + 1);
}
[TestMethod]
public void TestMethodPasses2()
{
StopAssert.AreEqual(2, 1 + 1);
}
}
[TestClass]
public class UnitTest2
{
[TestInitialize()]
public void Initialize()
{
StopAssert.IsFalse(StopAssert.Failed);
}
[TestMethod]
public void TestMethodPasses()
{
StopAssert.AreEqual(2, 1 + 1);
}
[TestMethod]
public void TestMethodFails()
{
StopAssert.AreEqual(0, 1 + 1);
}
[TestMethod]
public void TestMethodPasses2()
{
StopAssert.AreEqual(2, 1 + 1);
}
}
