Providing test data in Python

Question

How can I run the same test against a lot of different data?

I want to be reported of all failures.

For example:

def isEven(number):
    return True # Quite buggy implementation

data = [
    (2, True),
    (3, False),
    (4, True),
    (5, False),
]

class MyTest:
   def evenTest(self, num, expected):
       self.assertEquals(expected, isEven(num))

I have found solution which raises error on first failure only: PHPUnit style dataProvider in Python unit test

How can I run a test to be reported of all failures?

Possible duplicate: *[Python's unittest and dynamic creation of test cases](https://stackoverflow.com/questions/1193909)*. Which is itself a duplicate of *[How do you generate dynamic (parameterized) unit tests in Python?](https://stackoverflow.com/questions/32899)*. Why and why not? — Peter Mortensen, Feb 05 '21 at 15:35

score 12 · Answer 1 · edited Feb 05 '21 at 15:21

12

If you are using pytest you can go this way:

import pytest

def is_even(number):
    return True # Wuite buggy implementation

@pytest.mark.parametrize("number, expected", [
    (2, True),
    (3, False),
    (4, True),
    (5, False)
])
def test_is_even(number, expected):
    assert is_even(number) == expected

You will get something like (shortened):

/tmp/test_it.py:13: AssertionError
=========== 2 failed, 2 passed in 0.01 seconds ====================

edited Feb 05 '21 at 15:21

Peter Mortensen

30,738
21
105
131

answered Sep 22 '14 at 15:32

modesto

245
2
5

2

Maybe mark parametrize didn't exist when the other answers were contributed, but this feels like the correct answer here. – Craig Brett Sep 28 '17 at 08:41

Sven Marnach · Answer 2 · 2011-06-14T21:46:04.263

One solution is to make different test case instances for each entry in data:

class MyTest(unittest.TestCase):
    def __init__(self, num, expected):
        unittest.TestCase.__init__(self, "evenTest")
        self.num = num
        self.expected = expected
    def evenTest(self):
        self.assertEqual(self.expected, isEven(self.num))

For unittest to know how to construct the test cases, add a load_tests() function to your module:

def load_tests(loader, tests, pattern):
    return unittest.TestSuite(MyTest(num, expected)
                              for num, expected in data)

score 8 · Accepted Answer · edited Feb 05 '21 at 15:38

You should be using py.test. I think the unittest module was blindly copied from JUnit. Anyway, you can hack your way like this:

import unittest

data = [
    (2, True),
    (3, False),
    (4, True),
    (5, False)]

# This should be imported from a separate module.
def isEven(number):
    return True # Quite buggy implementation

def create_test_func(num, expected):
    def _test_func(self):
        self.assertEqual(expected, isEven(num))
    return _test_func

class TestIsEven(unittest.TestCase):

    pass

# pyunit isn't Pythonic enough. Use py.test instead
# till then we rely on such hackery
import new
for i, (num, expected) in enumerate(data):
    setattr(TestIsEven, 'test_data_%d'%i, create_test_func(num, expected))

if __name__ == "__main__":
    unittest.main()

And the output is:

.F.F
======================================================================
FAIL: test_data_1 (__main__.TestIsEven)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "untitled-1.py", line 15, in _test_func
    self.assertEqual(expected, isEven(num))
AssertionError: False != True

======================================================================
FAIL: test_data_3 (__main__.TestIsEven)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "untitled-1.py", line 15, in _test_func
    self.assertEqual(expected, isEven(num))
AssertionError: False != True

----------------------------------------------------------------------
Ran 4 tests in 0.000s

FAILED (failures=2)

Using this approach, you can add more niceties like printing debugging information on failure, etc.

+1, on `i think unittest module was blindly copied from junit` and it's not the only one that was copied from java-like :) — mouad, Jun 15 '11 at 07:13

score 5 · Answer 4 · edited Feb 05 '21 at 15:48

You may be looking for something like this:

import unittest


def is_even(number):
    return True # Quite buggy implementation


class TestCase(unittest.TestCase):
    def setUp(self):
        self.expected_output = [
            (2, True),
            (3, False),
            (4, True),
            (5, False)
        ]

    def test_is_even(self):
        real_res = []

        for arg, _ in self.expected_output:
            real_res.append((arg, is_even(arg)))

        msg_error = '\nFor %s Expected %s Got %s'
        msg = []
        for res1, res2 in zip(real_res, self.expected_output):
            if res1[1] != res2[1]:
                msg.append(msg_error % (res1[0], res1[1], res2[1]))


        self.assertEqual(real_res, self.expected_output, "".join(msg))


if __name__ == '__main__':
    unittest.main()

Output:

F
======================================================================
FAIL: test_is_even (__main__.TestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test.py", line 29, in test_example
    self.assertEqual(real_res, self.expected_output, ''.join(msg))
AssertionError:
For 3 Expected True Got False
For 5 Expected True Got False

----------------------------------------------------------------------
Ran 1 test in 0.000s

FAILED (failures=1)

If I had 50 tests, from which 10 fails it will be difficult to find what's going on. For small set of simple test data, it will be very handy and useful. Thanks :) — jinowolski, Jun 14 '11 at 22:08
@jinowolski: That true, gladly we can customize the error msg as much as we want i edited my answer i don't know if this work for you. — mouad, Jun 14 '11 at 22:21
@Anurag: Which is IMO what it should be, one (simple) function one test, and even if we want to split the tests at least they should all figure out in one Test Case, i can't imagine splitting this in more that one test case specially if this is just a part of a big package, but this is just my humble opinion :) — mouad, Jun 15 '11 at 07:09

score 0 · Answer 5 · edited Feb 05 '21 at 15:40

0

import unittest

data = [
    (2, True),
    (3, False),
    (4, True),
    (5, False)]

# This should be imported from a separate module.
def isEven(number):
    return True # Quite buggy implementation


class TestIsEven(unittest.TestCase):
    def test_is_even(self):
        for num, expected in data:
            self.assertEqual(expected, isEven(num))

edited Feb 05 '21 at 15:40

Peter Mortensen

30,738
21
105
131

answered Jun 14 '11 at 21:30

Steve Prentice

23,230
11
54
55

This raises on the first error -- read the end of the question again. – Sven Marnach Jun 14 '11 at 21:31

Providing test data in Python

5 Answers5