I'm attempting to use Pytest to write a dynamic test suite, where the test data is held in a separate file, e.g. a YAML file or a .csv. I want to run multiple tests, all of which are parameterised from the same file. Let's say I have a testing file test_foo.py
, that looks like this:
import pytest
@pytest.mark.parametrize("num1, num2, output", ([2, 2, 4], [3, 7, 10], [48, 52, 100]))
def test_addnums(num1, num2, output):
assert foo.addnums(num1, num2) == output
@pytest.mark.parametrize("foo, bar", ([1, 2], ['moo', 'mar'], [0.5, 3.14]))
def test_foobar(foo, bar):
assert type(foo) == type(bar)
Using the parametrize decorator, I can run multiple tests in pytest, and that works as expected:
test_foo.py::test_addnums[2-2-4] PASSED
test_foo.py::test_addnums[3-7-10] PASSED
test_foo.py::test_addnums[48-52-100] PASSED
test_foo.py::test_foobar[1-2] PASSED
test_foo.py::test_foobar[moo-mar] PASSED
test_foo.py::test_foobar[0.5-3.14] PASSED
But I want to parameterise these tests dynamically. By which I mean that, I want to write the test data for all tests in a separate file so that when I run pytest, it will apply all the test data I've written to each test function. Let's say I had a YAML file that looked something like:
test_addnums:
params: [num1, num2, output]
values:
- [2, 2, 4]
- [3, 7, 10]
- [48, 52, 100]
test_foobar:
params: [foo, bar]
values:
- [1, 2]
- [moo, mar]
- [0.5, 3.14]
I would then want to read this YAML file and use the data to parameterise all test functions in my test file.
I'm aware of the pytest_generate_tests
hook, and I've been trying to use this to load tests dynamically. I tried adding the same parameters and data values that I previously passed into the parametrize
decorator into the metafunc.parametrize
hook:
def pytest_generate_tests(metafunc):
metafunc.parametrize("num1, num2, output", ([2, 2, 4], [3, 7, 10], [48, 52, 100]))
metafunc.parametrize("foo, bar", ([1, 2], ['moo', 'mar'], [0.5, 3.14]))
def test_addnums(num1, num2, output):
assert foo.addnums(num1, num2) == output
def test_foobar(foo, bar):
assert type(foo) == type(bar)
This doesn't work, however, because pytest tries to apply the test data to every function:
collected 0 items / 1 error
=============================== ERRORS ================================
____________________ ERROR collecting test_foo.py _____________________
In test_addnums: function uses no argument 'foo'
======================= short test summary info =======================
ERROR test_foo.py
!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!
========================== 1 error in 0.16s ===========================
What I want to know is: how can I dynamically parameterise multiple tests using pytest? I've introspected pytest using pdb, and from what I can tell, metafunc
is only aware of the first test you've defined in the file. In my above example, test_addnums
is defined first, so when I print vars(metafunc)
in the pdb debugger, it shows these values:
(Pdb) pp vars(metafunc)
{'_arg2fixturedefs': {},
'_calls': [<_pytest.python.CallSpec2 object at 0x7f4330b6e860>,
<_pytest.python.CallSpec2 object at 0x7f4330b6e0b8>,
<_pytest.python.CallSpec2 object at 0x7f4330b6e908>],
'cls': None,
'config': <_pytest.config.Config object at 0x7f43310dbdd8>,
'definition': <FunctionDefinition test_addnums>,
'fixturenames': ['num1', 'num2', 'output'],
'function': <function test_addnums at 0x7f4330b5a6a8>,
'module': <module 'test_foo' from '<PATH>/test_foo.py'>}
But if I switch around the test_foobar
and test_addnums
functions, and reverse the order of the parametrize
calls, it shows information about test_foobar
instead.
(Pdb) pp vars(metafunc)
{'_arg2fixturedefs': {},
'_calls': [<_pytest.python.CallSpec2 object at 0x7f6d20d5e828>,
<_pytest.python.CallSpec2 object at 0x7f6d20d5e860>,
<_pytest.python.CallSpec2 object at 0x7f6d20d5e898>],
'cls': None,
'config': <_pytest.config.Config object at 0x7f6d212cbd68>,
'definition': <FunctionDefinition test_foobar>,
'fixturenames': ['foo', 'bar'],
'function': <function test_foobar at 0x7f6d20d4a6a8>,
'module': <module 'test_foo' from '<PATH>/test_foo.py'>}
So it seems like metafunc doesn't actually store information about every test function in my test file. Therefore I can't use fixturenames
or function
properties, as they only apply to one particular function, not all of them.
If that's the case, then how can I access all of the other test functions and parameterise them individually?