abstract test case using python unittest

Question

Is it possible to create an abstract TestCase, that will have some test_* methods, but this TestCase won't be called and those methods will only be used in subclasses? I think I am going to have one abstract TestCase in my test suite and it will be subclassed for a few different implementation of a single interface. This is why all test methods are the some, only one, internal method changes. How can I do it in elegant way?

Answer 1

I didn't quite understand what do you plan to do -- the rule of thumb is "not to be smart with tests" - just have them there, plain written.

But to achieve what you want, if you inherit from unittest.TestCase, whenever you call unittest.main() your "abstract" class will be executed - I think this is the situation you want to avoid.

Just do this: Create your "abstract" class inheriting from "object", not from TestCase. And for the actual "concrete" implementations, just use multiple inheritance: inherit from both unittest.TestCase and from your abstract class.

import unittest

class Abstract(object):
    def test_a(self):
        print "Running for class", self.__class__

class Test(Abstract, unittest.TestCase):
    pass

unittest.main()

update: reversed the inheritance order - Abstract first so that its defintions are not overriden by TestCase defaults, as well pointed in the comments bellow.

Answer 2

There's a very simple way that everyone has missed so far. And unlike several of the answers, it works with all test drivers, rather than failing the minute you switch between them.

Simply use inheritence as usual, then add:

del AbstractTestCase

at the end of the module. This will prevent the Abstract class from being executed.

Example:

import unittest
from abc import abstractmethod, ABC

class AbstractSomethingTest(ABC, unittest.TestCase):
   @abstractmethod
   def implement_me(self):
      pass

   def test_implemented_method_should_return_inri(self):
      self.assertEqual("INRI", self.implement_me())


class ImplementedSomethingTest(AbstractSomethingTest):
   def implement_me(self):
      return "INRI"

class AnotherClassImplementedSomethingTest(AbstractSomethingTest):
   def implement_me(self):
      return "INRI"

del AbstractSomethingTest

Answer 3

If you really want to use inheritance instead of mixins, a simple solution is to nest the abstract test in another class.

It avoids issues with test runner discovery and you can still import the abstract test from another module.

import unittest

class AbstractTests(object):
    class AbstractTest(unittest.TestCase)
        def test_a(self):
            print "Running for class", self.__class__

class Test(AbstractTests.AbstractTest):
    pass

Answer 4

Multiple inheritance isn't a great option here, chiefly for the two following reasons:

1. None of the methods in TestCase use super() so you'd have to list your class first for methods like setUp() and tearDown() to work.
2. pylint will warn that the base class uses self.assertEquals() etc which aren't defined on self at that point.

Here's the kludge I came up with: turn run() into a no-op for the base class only.

class TestBase( unittest.TestCase ):

  def __init__( self, *args, **kwargs ):
    super( TestBase, self ).__init__( *args, **kwargs )
    self.helper = None
    # Kludge alert: We want this class to carry test cases without being run
    # by the unit test framework, so the `run' method is overridden to do
    # nothing.  But in order for sub-classes to be able to do something when
    # run is invoked, the constructor will rebind `run' from TestCase.
    if self.__class__ != TestBase:
      # Rebind `run' from the parent class.
      self.run = unittest.TestCase.run.__get__( self, self.__class__ )                          
    else:
      self.run = lambda self, *args, **kwargs: None

  def newHelper( self ):
    raise NotImplementedError()

  def setUp( self ):
    print "shared for all subclasses"
    self.helper = self.newHelper()

  def testFoo( self ):
    print "shared for all subclasses"
    # test something with self.helper

class Test1( TestBase ):
  def newHelper( self ):
    return HelperObject1()

class Test2( TestBase ):
  def newHelper( self ):
    return HelperObject2()

Answer 5

Just to put in my two-cents, although it likely goes against some convention, you could define your abstract test case as a protected member to prevent its execution. I've implemented the following in Django and works as required. See example below.

from django.test import TestCase


class _AbstractTestCase(TestCase):

    """
    Abstract test case - should not be instantiated by the test runner.
    """

    def test_1(self):
        raise NotImplementedError()

    def test_2(self):
        raise NotImplementedError()


class TestCase1(_AbstractTestCase):

    """
    This test case will pass and fail.
    """

    def test_1(self):
        self.assertEqual(1 + 1, 2)


class TestCase2(_AbstractTestCase):

    """
    This test case will pass successfully.
    """

    def test_1(self):
        self.assertEqual(2 + 2, 4)

    def test_2(self):
        self.assertEqual(12 * 12, 144)

Answer 6

Raise unittest.SkipTest in setUpClass()

Just another approach is to raise a unittest.SkipTest in setUpClass() of the base class and override setUpClass() in child classes:

class BaseTestCase(TestCase):
    @classmethod
    def setUpClass(cls):
        "Child classes must override this method and define cls.x and cls.y"
        raise unittest.SkipTest

    def test_x(self):
        self.assertEqual(self.x * 3, self.x)

    def test_y(self):
        self.assertEqual(self.y * 3, self.y + self.y + self.y)

    def test_z(self):
        self.assertEqual(self.x + self.y, self.y)


class IntegerTestCase(BaseTestCase):
    @classmethod
    def setUpClass(cls):
        cls.x = 0
        cls.y = 2


class StringTestCase(BaseTestCase):
    @classmethod
    def setUpClass(cls):
        cls.x = ''
        cls.y = 'zuzuka'

If you need to use custom TestCase that defines its own setUpClass() and you need to call super().setUpClass(), you can define you own method to "set up data" and raise SkipTest only inside that method:

class BaseTestCase(ThidPartyTestCase):
    @classmethod
    def setUpClass(cls):
        super().setUpClass()  # if ThirdPartyTestCase has own setUpClass()
        cls.setUpTestCaseData()

    @classmethod
    def setUpTestCaseData(cls):
        "Override and set up cls.x and cls.y here"
        raise unittest.SkipTest

    ...  # tests


class IntegerTestCase(BaseTestCase):
    @classmethod
    def setUpTestCaseData(cls):
        cls.x = 0
        cls.y = 2

Answer 7

The unittest module provides several options for skipping tests.

My preferred solution is to override the setUpClass method in the "abstract" base class to raise a unittest.SkipTest exception if needed:

class BaseTestCase(unittest.TestCase):
  @classmethod
  def setUpClass(cls):
    if cls is BaseTestCase:
      raise unittest.SkipTest("%s is an abstract base class" % cls.__name__)
    else:
      super(BaseTestCase, cls).setUpClass()

Answer 8

Python unittest library has load_tests protocol, which can be used to achieve exactly what do you want:

# Add this function to module with AbstractTestCase class
def load_tests(loader, tests, _):
    result = []
    for test_case in tests:
        if type(test_case._tests[0]) is AbstractTestCase:
            continue
        result.append(test_case)
    return loader.suiteClass(result)

Answer 9

If you follow the convention of explicitly listing all test classes in run_unittest (see e.g. the Python test suite for many uses of that convention), then it will be straight-forward to not list a specific class.

If you want to continue using unittest.main, and if you can allow using unittest2 (e.g. from Python 2.7), you can use its load_tests protocol to specify which classes contain test cases). In earlier versions, you will have to subclass TestLoader, and override loadTestsFromModule.

Answer 10

Another reason for wanting to do what the OP is doing is to create a highly-parameterized base class which implements much of a set of core tests which need to be reproduced in several environments/scenarios. What I'm describing is essentially creating a parameterized fixture, a la pytest, using unittest.

Assuming you (like me) decide to run away as fast as you can from any multiple-inheritance-based solutions, one might have the following problem with using load_tests() to filter out your base class from the loaded suite:

In the standard TestLoader, load_tests is called after the auto-loading-from-class is done. Because: * this auto-loading-from-class will attempt to construct instances from your base class with the standard signature init(self, name), and * you may want this base class to have a very different ctor signature, or * you may want to skip construction-then-removal of your base class instances for some other reason

.. you may want to completely prevent this auto-loading of test instances from base classes.

EDIT: Vadim's solution in this other thread is a more elegant, concise, and independent way to do this. I have implemented the "nested class trick" and confirmed it works beautifully for the purpose of preventing TestLoader from "finding" your TestCase bases.

I originally had done this by modifying TestLoader.loadTestsFromModule to simply skip any TestCase classes which serve as base classes for any other TestCase classes in the module:

for name in dir(module):
    obj = getattr(module, name)
    # skip TestCase classes:
    # 1. without any test methods defined
    # 2. that are base classes
    #    (we don't allow instantiating TestCase base classes, which allows test designers
    #     to implement actual test methods in highly-parametrized base classes.)
    if isinstance(obj, type) and issubclass(obj, unittest.TestCase) and \
            self.getTestCaseNames(obj) and not isbase(obj, module):
        loaded_suite = self.loadTestsFromTestCase(obj)
        # ignore empty suites
        if loaded_suite.countTestCases():
            tests.append(loaded_suite)

where:

def isbase(cls, module):
    '''Returns True if cls is base class to any classes in module, else False.'''
    for name in dir(module):
        obj = getattr(module, name)
        if obj is not cls and isinstance(obj, type) and issubclass(obj, cls):
            return True
    return False

The parametrization I spoke of above is implemented by having each child class define it's fixture details (the parameters) and pass them to the base class TestCase ctor so that all of its common impl methods (the "fixturey" ones setUp*/tearDown*/cleanup* and the test methods themselves) have all the info that defines the now very specific fixture that that child TestCase class is to operate on.

For me, this was a temporary solution for quickly implementing some parametrized fixtures in unittest, since I plan to move my team's tests to pytest asap.

Answer 11

Here's a relatively simple approach that allows your common tests to inherit from TestCase (so type checking and IDE tooling stays happy), that uses only documented unittest features, and that avoids the "skip" test status:

import unittest

class CommonTestCases(unittest.TestCase):
    def __init__(self, methodName='runTest'):
        if self.__class__ is CommonTestCases:
            # don't run these tests on the abstract base implementation
            methodName = 'runNoTestsInBaseClass'
        super().__init__(methodName)

    def runNoTestsInBaseClass(self):
        print('not running tests in abstract base class')
        pass

    def test_common(self):
        # This will run *only* in subclasses. Presumably, this would 
        # be a test you need to repeat in several different contexts.
        self.assertEqual(2 + 2, 4)


class SomeTests(CommonTestCases):
    # inherited test_common *will* be run here

    def test_something(self):
        self.assertTrue(True)


# Also plays nicely with MRO, if needed:
class SomeOtherTests(CommonTestCases, django.test.SimpleTestCase):
    # inherited test_common *will* be run here

    def test_something_else(self):
        self.client.get('/')  # ...

How it works: per the unittest.TestCase documentation, "Each instance of TestCase will run a single base method: the method named methodName." The default "runTests" runs all the test* methods on the class—that's how TestCase instances normally work. But when running in the abstract base class itself, you can simply override that behavior with a method that does nothing.

A side effect is your test count will increase by one: the runNoTestsInBaseClass "test" gets counted as a successful test when it's run on CommonTestCases.

Answer 12

If you set __test__ = False in the base test class it should disable its tests. Quoting from this link:

class MessageTestBase(unittest.TestCase):
    __test__ = False

    def setUp(self):
        self.status = 'running'

    def tearDown(self):
        self.status = 'not running'

    def test_common_test(self):
        self.assertEqual(self.status, 'running')


class TestSlack(MessageTestMixin):
    __test__ = True 

Notice the differences. Our mixin becomes a base class that inherits from TestCase. we include __test__ = False in the base class to prevent the test runner from executing tests in this class. Then the child class only inherits from MessageTestBase and includes __test__ = True to instruct the test runner to run our tests.

More details here: How does __test__ = False magic attribute work for test discovery

Answer 13

I have done it following way, maybe it can inspire you:

class AbstractTest(TestCase):
    def setUp(self):
        pass

    def tearDown(self):
        pass

    def _test_1(self):
        # your test case here

class ConcreteTest(AbstractTest)

    def test_1(self):
        self._test_1()

Although it's not the most convenient solution, it lets you escape from multi inheritance. Also, the solution suggested by Dan Ward didn't work with Django testing in PyCharm.