How do you parametrize a pytest class with Hypothesis @given?

Question

In order to test how my database behaves when I add two very similar data rows, I need to setup a new database for each combination of parameters. I'm also using Hypothesis' strategies to generate "similar" data rows.

The test work flow should go as:

for example in hypothesis-given-examples:    # @given
    for combination in pytest-parametrized-combinations:   # @pytest.mark.parametrize
        db = setup_db(example, combination)  # should be a fixture with `yield` but I can't parametrize it
        do_test_1(db)  # using the setup database
        do_test_2(db)  # using the setup database
        do_test_3(db)  # using the setup database
        teardown(db)

I started with:

@mark.parametrize('different_properties', [False, 'entirekeys', 'crop', 'addkeys', 'overwritekeys'])
@mark.parametrize('different_identproperties', [False, 'entirekeys', 'crop', 'addkeys', 'overwritekeys'])
@mark.parametrize('different_labels', [False, 'entirekeys', 'crop', 'addkeys', 'overwritekeys'])
class TestMerge:
    @classmethod
    @given(node1=node_strategy, sampler=data())
    def setup_class(cls, node1, sampler, different_labels, different_properties, different_identproperties):
        node2 = create_node_from_node(node1, sampler, different_labels, different_properties,
                                      different_identproperties)  # uses hypothesis to generate
        cls.node1, cls.node2 = node1, node2

    def test_merge(self):
        read_node1, read_node2 = read(1), read(2)
        # do some test here comparing input and output

But it seems that is impossible to do a setup like this

So I tried to use a fixture with an indirect argument, but I want to do this for a whole class and copying the parameters for each test method seems ridiculous and also I can't make a cartesian product of parameter lists using indirect arguments that feed to one fixture.

Here is a minimal running/erroring example:

import pytest
from hypothesis import strategies as st
from hypothesis import given

@pytest.mark.parametrize('different_properties', [False, 'entirekeys', 'crop', 'addkeys', 'overwritekeys'], scope='class')
@pytest.mark.parametrize('different_identproperties', [False, 'entirekeys', 'crop', 'addkeys', 'overwritekeys'], scope='class')
@pytest.mark.parametrize('different_labels', [False, 'entirekeys', 'crop', 'addkeys', 'overwritekeys'], scope='class')
class TestMerge:
    @classmethod
    @given(node1=st.integers(), sampler=st.data())
    def setup_class(cls, node1, sampler, different_labels, different_properties, different_identproperties):
        node2 = node1 * 2  # just for the minimal example
        cls.node1, cls.node2 = node1, node2

    def test_merge(self, different_labels, different_properties, different_identproperties):
        # read_node1, read_node2 = read(1), read(2)
        # do some test here comparing input and output
        pass

Pytest output:

ERROR test_minimal_example.py::TestMerge::test_merge[False-False-False] - TypeError: setup_class() missing 3 required...
ERROR test_minimal_example.py::TestMerge::test_merge[False-False-entirekeys] - TypeError: setup_class() missing 3 req...
    ...

Hopefully you can see what I'm trying to do. If not, please ask!

Thanks!

Please provide a [mcve]. The code you provided right now is erroneous (e.g. there's no `self` in classmethods). — hoefling, Dec 15 '20 at 13:08
That's still not runnable without imports - please provide a *complete* reproducible example, if we can't copy-paste it into a file and get the same result it's not reproducible. — Zac Hatfield-Dodds, Dec 16 '20 at 03:03
As you have found you can't use pytest.mark.parametrize with class_setup function. Can you not use Hypothesis to generate these combinations instead? — Chris, Feb 12 '21 at 18:23

score 1 · Answer 1 · answered Dec 16 '20 at 03:05

1

It's not clear why you're using a class here. Try e.g.

@pytest.mark.parametrize("p1", [True, False])
@pytest.mark.parametrize("p2", [True, False])
@given(ex=st.integers())
def test(p1, p2, ex):
    with setup_db(ex, (p1, p2)) as db:
        do_test_1(db)
        do_test_2(db)
        do_test_3(db)

(and note that this also works for test methods, if you add a self in the appropriate spot)

answered Dec 16 '20 at 03:05

Zac Hatfield-Dodds

2,455
6
19

It’s because I want to run more than one test. It’s for exactly the same reason that we use class scoped tiled fixtures. I want to set up the database once and then distinguish 10 or more separate tests on that database before destroying it. I want the tests to be separate so I can debug properly. – Lucidnonsense Dec 16 '20 at 07:23
But if this is the only way then I will do this. I just find it strange that you can’t parameterize a class with given – Lucidnonsense Dec 16 '20 at 07:27
Unfortunately there's no clear interpretation of what 'parametrizing a class with given' would actually do - I think you're imagining that we'd automatically wrap each method to supply the shared arguments. However (a) this would inevitably work for some test runners but not others, and (b) in many cases you'd want to supply other arguments with a second (method-level) given decorator but it's invalid to apply given twice. – Zac Hatfield-Dodds Jan 14 '21 at 03:56

How do you parametrize a pytest class with Hypothesis @given?

1 Answers1