1

Currently my app has a somewhat unstable development environment and a few of my tests will result in a false negative. If one of my tests fails in my Subliminal test suite, the entire suite stops! This is not what I want and I dont want to do something like surrounding my tests in try/catch blocks so how do I prevent my Subliminal test suite from stopping after one fail?

Ray
  • 501
  • 4
  • 9
  • Is the failure in the `setUpTest` or `tearDownTest` or `setUpTestCaseWithSelector:` or `tearDownTestCaseWithSelector:` methods, or is it actually in one of the test case methods? Is the failure in an `SLAssert` or is it an "unexpected error" where the test case throws an exception separate from any Subliminal assertion? Under normal circumstances a test case's failure should not stop the rest of the test cases from executing. – Aaron Golden Nov 13 '13 at 19:35
  • You are correct. I found out that I was getting an unexpected error and that caused the test suite to halt. – Ray Dec 02 '13 at 23:35

1 Answers1

1

There are two scenarios in which an exception thrown as part of a Subliminal test suite will cause the whole suite to stop. These are:

  1. If the exception occurs inside of the setUpTest method, which runs before the test cases in the test suite begin executing. In this case none of the test cases will execute.
  2. If the exception is "unexpected" meaning that it occurs outside of any SLAssert macro. Expected exceptions are the ones that the test case (or setUpTest for that matter) actually look for via SLAssert calls. Any other exceptions are considered to be essentially bugs in the test suite/case itself---something went wrong that the tests weren't prepared to handle. So the suite stops at that point instead of continuing with the app in a state it might not be able to handle.

From your comment it sounds like you ran into the second case. You could probably get the tests to continue by adding additional SLAssert calls to look for the unexpected issue that you're seeing (or by resolving whatever issue caused the exception in the first place).

Aaron Golden
  • 7,092
  • 1
  • 25
  • 31