How to manage the expectations of a test that uses a fixture with parameters

Question

Pytest gives you the ability to parameterize fixtures:

@pytest.fixture(params = ['a'])
def root(request):
    return request.param

so now in a text marked with our fixture "root" we can use our param:

def test_something(root):
    assert root == 'a' #this will pass

We can extend our fixture with another fixture:

@pytest.fixture(params = ['a'])
def root(request, leaf):
    return request.param + leaf


@pytest.fixture(params = ['b', 'c'])
def root(request):
    return request.param

Now that we have done this, its not clear how do manage our tests expectations.

To be clear, but test expectations i mean...

def test_something(root):
    assert root == expectation??? <---- this has to be hardcoded.

as a toy solution for this problem we could do

def test_something(root):
    assert root == {"ac": "ac", "ab": "ab"}[leaf]

But that's horrible for a host of obvious reasons. At EuroCon 2014, their was a talk on Advanced used of pytest fixtures which utilizes a fixture with params that uses a another fixture with params (like in my toy example). However the presenters expectation is simple:

assert inst.it_works

Which seems to skirt the issues.

The issue being that (in my example) at this point we have created a 'testing tree' with 'a' at the root and 'ab' and 'ac' as the leaves:

       'a'        
+-----+   +-----+ 
|               | 
|               | 
+>             <+ 

ab              ac

and while in this case its easy to keep track of the final state (the leaves) , it needs to be effortless, obvious and upfront (flat) in order for it to work.

The idea of a testing tool that could accommodate allowing you to use nested fixtures with parameters this way seems very powerful:

It would allow for easily testing all relevant states. Where in a typical testing suite you would need to write all the tests by hand, here they would be generated for you.
It would allow for pin pointing where in the tree failures were happening, allowing you to pinpoint the fixture that was possible the root cause of the problem.

So i'm curious if a tool exists, in python or another language that provides this functionality, while allowing us to manage our testing expectations.

What would such a tool look like? Well, my guess is that it would have to provide the user with a way to input expectations to a tests. Something like

> "test_ab_cd)
> inspect info("test_ab_cd")
> ab == ???
> ac == ???
> some testing info
> "ab" === ???
> enter what "ab" should equal?
> some information about the test
> "ac" === ???
> inspect info("test_ab_cd")
> ab == ab
> ac == ???

And additional provide logs and a testing file for easy reading.. something like ...

test_something.txt
=========================
- test_something_ab
assert ab == ab
- test_someting_ac
assert ac == ac

which in a more complex example would need to look something like this (essential what regular tests look like)

test_something.txt
====================
- test_something_graph
x = 1
y = 2
start = node(x, y)
que = SimplePriority()
finish_node == graph(start,que) 
assert finish_node == 5

Which would have to be generated via inspection of the various text_fixtures involved.

Its more then likely the solution here is to just have damp tests and not deeply nested fixtures. But i'm curious if there is a way to manage the expectations dynamically. Or maybe if there are any examples in other languages of such a testing tool. I searched around but didn't find anything.

fixtures are state before the test action, in your example: def test_something(root): assert root == 'a' #this will pass this doesn't make any sense to me your test should do some action with the state, and then assert new state and for assertion you can of course use any fixtures as well: def test_something(root, leaf): assert calculate(root) == leaf — Anatoly Bubenkov, Mar 27 '15 at 10:27
@bubenkoff , this is a toy example, just a bare bones representation of the problem. — Drew Verlee, Mar 27 '15 at 14:30

score 1 · Answer 1 · answered Jan 18 '16 at 05:07

OK so... in my opinion if you have different test result expectations for different input, then you have data test parametrization, rather than a fixture. Parametrization and fixture have a lot of overlap and so it can be tricky to figure out when to use one or the other. But when your results vary according to input, that shouts 'parametrize' to me. Result should be the same for all input? That shouts 'fixture'.

We can mix and match, though:

import pytest


def something(root, leaf):
    return root + leaf


@pytest.fixture(params=['a'])
def root(request):
    return request.param


@pytest.mark.parametrize('leaf,expected', [
    ('b', 'ab'),
    ('c', 'ac'),
])
def test_something(root, leaf, expected):
    assert something(root, leaf) == expected

But if you have multiple values for root, you're going to need to spell them out explicitly in parametrize, along with the expected values. You can only vary one thing at a time and still have readable tests.

How to manage the expectations of a test that uses a fixture with parameters

1 Answers1