How to split python 3 unittests into separate file with script to control which one to run?

Question

I want to split my Python 3.4 unit tests in separate modules and still be able to control which tests to run or skip from the command line, as if all tests were located in the same file. I'm having trouble doing so.

According to the docs, command line arguments can be used to select which tests to run. For example:

TestSeqFunc.py:

#!/usr/bin/env python3
# -*- coding: utf-8 -*-


import random
import unittest

class TestSequenceFunctions(unittest.TestCase):

    def setUp(self):
        self.seq = list(range(10))

    def test_shuffle(self):
        # make sure the shuffled sequence does not lose any elements
        random.shuffle(self.seq)
        self.seq.sort()
        self.assertEqual(self.seq, list(range(10)))

        # should raise an exception for an immutable sequence
        self.assertRaises(TypeError, random.shuffle, (1,2,3))

    def test_choice(self):
        element = random.choice(self.seq)
        self.assertTrue(element in self.seq)

    def test_sample(self):
        with self.assertRaises(ValueError):
            random.sample(self.seq, 20)
        for element in random.sample(self.seq, 5):
            self.assertTrue(element in self.seq)

if __name__ == '__main__':
    unittest.main()

can be controlled with either:

./TestSeqFunc.py

to run all tests in the file,

./TestSeqFunc.py TestSequenceFunctions

to run all tests defined in the TestSequenceFunctions class, and finally:

./TestSeqFunc.py TestSequenceFunctions.test_sample

to run the specific test_sample() method.

The problem I have is that I cannot find an organization of files that will allow me to:

1. Have multiple modules containing multiple classes and methods in separate files
2. Use a kind of wrapper script that will give the same kind of control over which tests (module/file, class, method) to run.

The problem I have is I cannot find a way to emulate the python3 -m unittest behaviour using a run_tests.py script. For example, I want to be able to do:

1. Run all the tests in the current directory So ./run_tests.py -v should do do the same as python3 -m unittest -v
2. Run one module (file): ./run_tests.py -v TestSeqFunc being equivalent to python3 -m unittest -v TestSeqFunc
3. Run one class: ./run_tests.py -v TestSeqFunc.TestSequenceFunctions being equivalent to python3 -m unittest -v TestSeqFunc.TestSequenceFunctions
4. Run specific methods from a class: ./run_tests.py -v TestSeqFunc.TestSequenceFunctions.test_sample being equivalent to python3 -m unittest -v TestSeqFunc.TestSequenceFunctions.test_sample

Note that I want to:

1. be able to pass arguments to unittests, for example the verbose flag used previously;
2. allow running specific modules, classes and even methods.

As of now, I use a suite() function in my run_all.py script which loads manually the modules and add their class to a suite using addTest(unittest.makeSuite(obj)). Then, my main() is simple:

if __name__ == '__main__':
    unittest.main(defaultTest='suite')

But using this I cannot run specific tests. In the end, I might just execute python3 -m unittest <sys.argv> from inside the run_all.py script, but that would be inelegant...

Any suggestions?!

Thanks!

Answer 1

Here's my final run_all.py:

#!/usr/bin/env python3
# -*- coding: utf-8 -*-

import unittest
import glob

test_pattern = 'validate_*.py'

if __name__ == '__main__':

    # Find all files matching pattern
    module_files = sorted(glob.glob(test_pattern))
    module_names = [os.path.splitext(os.path.basename(module_file))[0] for module_file in module_files]

    # Iterate over the found files
    print('Importing:')
    for module in module_names:
        print('    ', module)
        exec('import %s' % module)

    print('Done!')
    print()

    unittest.main(defaultTest=module_names)

Notes:

1. I use exec() to simulate 'import modulename'. The issue is that using importlib (explained here for example) will import the module but will not create a namespace for the module content. When I type import os, an "os" namespace is created and I can then access os.path. By using importlib, I couldn't figure out a way to do create that namespace. Having such a namespace is required for unittest; you get these kind of errors:

   Traceback (most recent call last):
   File "./run_all.py", line 89, in <module>
       unittest.main(argv=sys.argv)
   File "~/usr/lib/python3.4/unittest/main.py", line 92, in __init__
       self.parseArgs(argv)
   File "~/usr/lib/python3.4/unittest/main.py", line 139, in parseArgs
       self.createTests()
   File "~/usr/lib/python3.4/unittest/main.py", line 146, in createTests
       self.module)
   File "~/usr/lib/python3.4/unittest/loader.py", line 146, in loadTestsFromNames
       suites = [self.loadTestsFromName(name, module) for name in names]
   File "~/usr/lib/python3.4/unittest/loader.py", line 146, in <listcomp>
       suites = [self.loadTestsFromName(name, module) for name in names]
   File "~/usr/lib/python3.4/unittest/loader.py", line 114, in loadTestsFromName
       parent, obj = obj, getattr(obj, part)
   AttributeError: 'module' object has no attribute 'validate_module1'
   

   Hence the use of exec().

2. I have to add defaultTest=module_names or else main() defaults to all test classes inside the current file. Since there is no test class in run_all.py, nothing gets executed. So defaultTest must point to a list of all the modules name.

Answer 2

You can pass command-line arguments to unittest.main using the argv parameter:

The argv argument can be a list of options passed to the program, with the first element being the program name. If not specified or None, the values of sys.argv are used. (my emphasis)

So you should be able to use

if __name__ == '__main__':
    unittest.main(defaultTest='suite')

without any change and be able to call your script with command-line arguments as desired.