Is it possible to make a sub-test "expected failure" in Python unittest?

Question

In Python's unit test framework it is possible to declare that a test is expected to fail using the expectedFailure decorator. I use 'expected failures' to denote bugs in the implementation that need to be fixed at some later time, but are not critical. However, the expcetedFailure decorator applies to the whole test. I was wondering if there is a way to declare that a sub-test is expected to fail?

For example, consider the following test for some is_prime routine that does not declare 2 as a prime number (perhaps because it first filters out all even numbers):

class NumbersTest(unittest.TestCase):
    def test_primes(self):
        for i in range(2, 13):
            with self.subTest(i=i):
                if i in [2, 3, 5, 7, 11]:
                    self.assertTrue(is_prime(i))
                else:
                    self.assertFalse(is_prime(i))

The test obviously fails for i=2 because of the bug in is_prime, but as I am sure that for the now I will never call is_prime on 2, I would not make a big fuss out of it for the moment. I would like to have a means to declare that only the sub test for i=2 is expected to fail. If I decorate the entire test_primes with @expectedFailure then the routine does not get tested at all, so if someone changes the implementation and breaks its functionality it will not be noticed.

One possiblity is to use skipTest instead:

class NumbersTest(unittest.TestCase):
    def test_primes(self):
        for i in range(2, 13):
            with self.subTest(i=i):
                if i == 2:
                    self.skipTest("`is_prime` does not handle 2 correctly")
                if i in [2, 3, 5, 7, 11]:
                    self.assertTrue(is_prime(i))
                else:
                    self.assertFalse(is_prime(i))

But I am not too happy with skipping the test, as a test should normally be skipped if it cannot be performed (e.g., because of unavailable resources, etc.). In this case nothing prohibits us from running the test for i==2, it only fails and the failure is a known bug.

Why not use something like https://stackoverflow.com/questions/129507/how-do-you-test-that-a-python-function-throws-an-exception — Tin Nguyen, Mar 27 '20 at 09:30
@TinNguyen Could you please elaborate more? How the proposed solution could be applied to this problem? — MikeL, Mar 27 '20 at 09:33
You could split your test in two: Have `test_primes():` which handles all the cases which must be seen to work and then have `@unittest.expectedFailure def test_not_working_primes():` which are actually expected to fail at this time. Then when you get round to fixing bugs, the tests will alert you to move the previous failures into `test_primes()`. — quamrana, Mar 27 '20 at 09:59
Another option is to use [ApprovalTests](https://github.com/approvals/ApprovalTests.Python) and use `verify([is_prime(i) for i in range(2,13)])` to produce a result that you manually verify. Then when you get round to fixing bugs you can manually verify the changes. — quamrana, Mar 27 '20 at 10:05
@quamrana I thought about splitting the tests, which would be my last resort if there is indeed no way to declare expected failure on sub tests. This is not ideal, however, because (i) I would need to repeat many lines of test (in my real application the test is more complicated than this MWE), and (ii) once `is_prime` is corrected I would have to merge back two identical tests — MikeL, Mar 27 '20 at 10:06

score 4 · Answer 1 · answered Dec 06 '20 at 15:09

I realize this is kind of old, but if you can refactor the test to a method that accepts inputs, you can achieve this with minimal duplication

class NumbersTest(unittest.TestCase):
    def test_primes(self):
        for i in range(2, 13):
            with self.subTest(i=i):
                if i in [2, 3, 5, 7, 11]:
                    self.assertTrue(is_prime(i))
                else:
                    self.assertFalse(is_prime(i))

becomes

class NumbersTest(unittest.TestCase):
    def _test_primes(self, i, expectation):
       with self.subTest(i=i, expectation=expectation):
             self.assertEqual(is_prime(i), expectation)

    @unittest.expectedFailure
    def test_failing_primes(self):
         values = [2]
         for i in values:
             self._test_primes(i, False)

    def test_primes(self):
         for i in range(3, 13):
            if i in [3, 5, 7, 11]:
                self._test_primes(i, True)
            else:
                self._test_primes(i, False)

score 0 · Answer 2 · answered Jul 19 '21 at 12:56

I had the same wish for marking subtests as expected failures. This is especially relevant to me because my setUp/tearDown sets up a new virtual machine (qemu or dosemu2) with the application for each test which usually takes longer than a second, so bundling many tests as subtests helps keep the time down.

I went with the following code:

        for (msg, inst, code, success) in asmtests:
            with self.subTest(msg = msg + ": " + inst):
                actuallysucceeded = 0
                try:
                    # do the test
                    actuallysucceeded = 1
                except AssertionError as e:
                    if success:
                        raise e
                    else:
                        pass
                if actuallysucceeded and not success:
                    raise AssertionError("unexpected success")

If the test raises an AssertionError (which seems to be what unittest's asserts do on failures) we check the success flag: If it is set (expected success) we simply re-raise the exception for unittest to handle. Otherwise we ignore this exception. If the actuallysucceeded flag is set afterwards and the success flag is not set then we manually raise an AssertionError.

This program logic negates the sense of an expected failure subtest and allows other subtests to work as expected using the same handling.

Is it possible to make a sub-test "expected failure" in Python unittest?

2 Answers2