Python unittest: cancel all tests if a specific test fails

Question

I am using unittest to test my Flask application, and nose to actually run the tests.

My first set of tests is to ensure the testing environment is clean and prevent running the tests on the Flask app's configured database. I'm confident that I've set up the test environment cleanly, but I'd like some assurance of that without running all the tests.

import unittest

class MyTestCase(unittest.TestCase):
    def setUp(self):
        # set some stuff up
        pass

    def tearDown(self):
        # do the teardown
        pass

class TestEnvironmentTest(MyTestCase):
    def test_environment_is_clean(self):
        # A failing test
        assert 0 == 1

class SomeOtherTest(MyTestCase):
    def test_foo(self):
        # A passing test
        assert 1 == 1

I'd like the TestEnvironmentTest to cause unittest or noseto bail if it fails, and prevent SomeOtherTest and any further tests from running. Is there some built-in method of doing so in either unittest (preferred) or nose that allows for that?

Do you know if nose solves the test-ordering issue? Because in python you're not guaranteed test order (though I think it usually resolves to alphabetical). — Colleen, Oct 15 '12 at 18:14
Well that's going to be another problem, then, too... but I suppose that's solvable by putting the test in your setup. — Colleen, Oct 15 '12 at 18:18
http://stackoverflow.com/questions/6813837/stop-testsuite-if-a-testcase-find-an-error ? — Colleen, Oct 15 '12 at 18:19
@Colleen hmm not quite... I want to run all tests even if there are errors with them, but I want my environment tests to pass prior to any other test running. Your first suggestion works though: I dropped the environment test into setUp. It seems like I was wrong... something is persisting between tests, bad bad bad! — Colin Dunklau, Oct 15 '12 at 18:48
Oh, sorry, misunderstood your question. Thought you wanted to fail immediately if you failed the TestEnvironmentTest. — Colleen, Oct 15 '12 at 18:49
@Colleen I've updated my answer. Drop the updated code in an answer, and I'll accept it in a few days if nobody has a better solution. — Colin Dunklau, Oct 15 '12 at 18:55
One can overcome the issue of test case execution order by naming the tests in a way that they line up in ascending manner. Eg: First test named: `test_1000_xxxx`, Second test named: `test_1100_xxxx` and so on On unittest documentation it mentions: `Note that the order in which the various test cases will be run is determined by sorting the test function names with respect to the built-in ordering for strings.` URL: [link]https://docs.python.org/2/library/unittest.html — Deep, Dec 19 '14 at 08:39

score 10 · Accepted Answer · edited Feb 28 '18 at 15:16

In order to get one test to execute first and only halt execution of the other tests in case of an error with that test, you'll need to put a call to the test in setUp() (because python does not guarantee test order) and then fail or skip the rest on failure.

I like skipTest() because it actually doesn't run the other tests whereas raising an exception seems to still attempt to run the tests.

def setUp(self):
    # set some stuff up
    self.environment_is_clean()

def environment_is_clean(self):
    try:
        # A failing test
        assert 0 == 1
    except AssertionError:
        self.skipTest("Test environment is not clean!")

score 4 · Answer 2 · answered Oct 15 '12 at 21:09

4

For your use case there's setUpModule() function:

If an exception is raised in a setUpModule then none of the tests in the module will be run and the tearDownModule will not be run. If the exception is a SkipTest exception then the module will be reported as having been skipped instead of as an error.

Test your environment inside this function.

answered Oct 15 '12 at 21:09

Piotr Dobrogost

41,292
40
236
366

Or in some cases, `setUpClass()` will also work `If an exception is raised during a setUpClass then the tests in the class are not run and the tearDownClass is not run.` http://docs.python.org/2/library/unittest.html#setupclass-and-teardownclass – Dennis Golomazov Jul 23 '13 at 08:59
If `setUpModule()` is defined as a function outside the `TestCase` class, how would you make an assertion? – Stevoisiak Feb 28 '18 at 14:37

score 2 · Answer 3 · answered Oct 15 '12 at 19:01

2

You can skip entire test cases by calling skipTest() in setUp(). This is a new feature in Python 2.7. Instead of failing the tests, it will simply skip them all.

answered Oct 15 '12 at 19:01

kichik

33,220
7
94
114

That's a nice feature, but the final result is OK, when it should whine about errors. – Colin Dunklau Oct 15 '12 at 19:08
That's what you have TestEnvironmentTest failure for. If you just want them to fail, what's wrong with the code you already have? – kichik Oct 15 '12 at 20:42

score 1 · Answer 4 · answered Oct 15 '12 at 19:03

I'm not quite sure whether it fits your needs, but you can make the execution of a second suite of unittests conditional on the result of a first suite of unittests:

envsuite = unittest.TestSuite()
moretests = unittest.TestSuite()
# fill suites with test cases ...
envresult = unittest.TextTestRunner().run(envsuite)
if envresult.wasSuccessful():
    unittest.TextTestRunner().run(moretests)

Python unittest: cancel all tests if a specific test fails

4 Answers4

Linked