How does Python's unittest module detect test cases?

Question

I was wondering when we run unittest.main(), how does Python know what subclasses unittest.Testcase has?

For example, if I add a class FromRomanBadInput(unittest.TestCase), how does unittest know to run this?

Answer 1

So I looked around in my Python27/Lib directory...

unittest.main is actually an alias for a class, unittest.TestProgram. So what happens is you construct an instance of this, and its __init__ runs, which does a bunch of sanity checks and configuration, including a dynamic import of the module that you called it from (it uses the __import__ function, with __main__ as the name of the module to import, by default). So now it has a self.module attribute that contains a module object that represents your source.

Eventually, it gets to this code:

self.test = self.testLoader.loadTestsFromModule(self.module)

where self.testLoader is an instance of unittest.TestLoader. That method contains, among other stuff:

    for name in dir(module):
        obj = getattr(module, name)
        if isinstance(obj, type) and issubclass(obj, case.TestCase):
            tests.append(self.loadTestsFromTestCase(obj))

So it uses the dir of your module object to get the names of all the global variables you defined (including classes), filters that to just the classes that derive from unittest.TestCase (locally, case.TestCase is an alias for that), and then looks for test methods inside those classes to add to the tests list. That search behaves similarly:

    def isTestMethod(attrname, testCaseClass=testCaseClass,
                     prefix=self.testMethodPrefix):
        return attrname.startswith(prefix) and \
            hasattr(getattr(testCaseClass, attrname), '__call__')
    testFnNames = filter(isTestMethod, dir(testCaseClass))

so it uses the dir of the class to get a list of names to try, looks for attributes with those names, and selects those that start with the self.testMethodPrefix ('test' by default) and that are callable (have, in turn, a __call__ attribute). (I'm actually surprised they don't use the built-in callable function here. I guess this is to avoid picking up nested classes.)

Answer 2

the 'main' function searches for all classes which inherits the unittest.TestCase in imported modules. and current path, then tries to run each method that starts with 'test'

from python's document:

import random
import unittest

class TestSequenceFunctions(unittest.TestCase):

    def setUp(self):
        self.seq = range(10)

    def test_shuffle(self):
        # make sure the shuffled sequence does not lose any elements
        random.shuffle(self.seq)
        self.seq.sort()
        self.assertEqual(self.seq, range(10))

        # should raise an exception for an immutable sequence
        self.assertRaises(TypeError, random.shuffle, (1,2,3))

    def test_choice(self):
        element = random.choice(self.seq)
        self.assertTrue(element in self.seq)

    def test_sample(self):
        with self.assertRaises(ValueError):
            random.sample(self.seq, 20)
        for element in random.sample(self.seq, 5):
            self.assertTrue(element in self.seq)

if __name__ == '__main__':
    unittest.main()

A testcase is created by subclassing unittest.TestCase. The three individual tests are defined with methods whose names start with the letters test. This naming convention informs the test runner about which methods represent tests.

Answer 3

I wrote some code that attempts to do behave similarly to unittest.main() below. In summary, I iterate through the modules, and for the modules that don't start with the name 'unittest', I inspect its members. Then if those members is a class and is a subclass of unittest.TestCase, I parse through that class' members. Then if those class' members is a function or method that starts with 'test', I add it to the list of tests. The class object's __dict__ is used to introspect the methods/functions since using inspect.getmembers may show too much. Finally that list of tests is converted to a tuple and wrapped up as a suite. Then the suite is ran using the runner at verbosity level 2. Note that, of course, removing the regex that checks for 'test' at the beginning of a function/method name can be removed to include bar_test() to the list of tests if you don't want that restriction.

#!/usr/bin/env python

import unittest
import inspect
import sys
import re

class Foo(unittest.TestCase):
   @staticmethod
   def test_baz():
      pass

   @classmethod
   def test_mu(cls):
      pass

   def test_foo(self):
      self.assertEqual('foo', 'foo')

   def bar_test(self):
      self.assertEqual('bar', 'bar')

class Bar:
   pass

if __name__ == '__main__':
   runner = unittest.TextTestRunner(verbosity=2)
   tests = []
   is_member_valid_test_class = lambda member: inspect.isclass(member) and \
      issubclass(member, unittest.TestCase)

   for module_name, module_obj in sys.modules.items():
      if not re.match(r'unittest', module_name):
         for cls_name, cls in inspect.getmembers(
            module_obj, is_member_valid_test_class):
            for methname, methobj in cls.__dict__.items():
               if inspect.isroutine(methobj) and re.match(r'test', methname):
                  tests.append(cls(methname))

   suite = unittest.TestSuite(tests=tuple(tests))
   runner.run(suite)

The resulting output is:

test_foo (__main__.Foo) ... ok
test_baz (__main__.Foo) ... ok
test_mu (__main__.Foo) ... ok

----------------------------------------------------------------------
Ran 3 tests in 0.001s

OK