Separate test cases per input files?

Question

Most test frameworks assume that "1 test = 1 Python method/function", and consider a test as passed when the function executes without raising assertions.

I'm testing a compiler-like program (a program that reads *.foo files and process their contents), for which I want to execute the same test on many input (*.foo) files. IOW, my test looks like:

class Test(unittest.TestCase):
    def one_file(self, filename):
        # do the actual test

    def list_testcases(self):
        # essentially os.listdir('tests/') and filter *.foo files.

    def test_all(self):
        for f in self.list_testcases():
            one_file(f)

My current code uses unittest from Python's standard library, i.e. one_file uses self.assert...(...) statements to check whether the test passes.

This works, in the sense that I do get a program which succeeds/fails when my code is OK/buggy, but I'm loosing a lot of the advantages of the testing framework:

• I don't get relevant reporting like "X failures out of Y tests" nor the list of passed/failed tests. (I'm planning to use such system not only to test my own development but also to grade student's code as a teacher, so reporting is important for me)

• I don't get test independence. The second test runs on the environment left by the first, and so on. The first failure stops the testsuite: testcases coming after a failure are not ran at all.

• I get the feeling that I'm abusing my test framework: there's only one test function so automatic test discovery of unittest sounds overkill for example. The same code could (should?) be written in plain Python with a basic assert.

An obvious alternative is to change my code to something like

class Test(unittest.TestCase):
    def one_file(self, filename):
        # do the actual test

    def test_file1(self):
        one_file("first-testcase.foo")

    def test_file2(self):
        one_file("second-testcase.foo")

Then I get all the advantages of unittest back, but:

• It's a lot more code to write.

• It's easy to "forget" a testcase, i.e. create a test file in tests/ and forget to add it to the Python test.

I can imagine a solution where I would generate one method per testcase dynamically (along the lines of setattr(self, 'test_file' + str(n), ...)), to generate the code for the second solution without having to write it by hand. But that sounds really overkill for a use-case which doesn't seem so complex.

How could I get the best of both, i.e. automatic testcase discovery (list tests/*.foo files), test independence and proper reporting?

Answer 1

If you can use pytest as your test runner, then this is actually pretty straightforward using the parametrize decorator:

import pytest, glob

all_files = glob.glob('some/path/*.foo')

@pytest.mark.parametrize('filename', all_files)
def test_one_file(filename):
    # do the actual test

This will also automatically name the tests in a useful way, so that you can see which files have failed:

$ py.test
================================== test session starts ===================================
platform darwin -- Python 3.6.1, pytest-3.1.3, py-1.4.34, pluggy-0.4.0
[...]
======================================== FAILURES ========================================
_____________________________ test_one_file[some/path/a.foo] _____________________________

filename = 'some/path/a.foo'

    @pytest.mark.parametrize('filename', all_files)
    def test_one_file(filename):
>      assert False
E      assert False

test_it.py:7: AssertionError
_____________________________ test_one_file[some/path/b.foo] _____________________________

filename = 'some/path/b.foo'

    @pytest.mark.parametrize('filename', all_files)
    def test_one_file(filename):
[...]

Answer 2

Here is a solution, although it might be considered not very beautiful... The idea is to dynamically create new functions, add them to the test class, and use the function names as arguments (e.g., filenames):

# import
import unittest

# test class
class Test(unittest.TestCase):

    # example test case
    def test_default(self):
        print('test_default')
        self.assertEqual(2,2)

# set string for creating new function    
func_string="""def test(cls):

        # get function name and use it to pass information
        filename = inspect.stack()[0][3]

        # print function name for demonstration purposes
        print(filename)

        # dummy test for demonstration purposes
        cls.assertEqual(type(filename),str)"""

# add new test for each item in list
for f in ['test_bla','test_blu','test_bli']:

    # set name of new function
    name=func_string.replace('test',f)

    # create new function
    exec(name)

    # add new function to test class
    setattr(Test, f, eval(f))

if __name__ == "__main__":
    unittest.main()

This correctly runs all four tests and returns:

> test_bla
> test_bli
> test_blu
> test_default
> Ran 4 tests in 0.040s
> OK