pytest new test for every iteration | for loop | parametrize fixture

Question

Basically, I'm trying to do a test for each iteration of a list of routes to check that the web pages associated with a particular function return a valid status code.

I want something along the lines of this:

import pytest
from flask import url_for
from myflaskapp import get_app

@pytest.yield_fixture
def app():
    # App context and specific overrides for test env
    yield get_app()

@pytest.yield_fixture
def client(app):
    yield app.test_client()

@pytest.yield_fixture
def routes(app):
    routes = [
    'foo',
    'bar',
    # There's quite a lot of function names here
    ]
    with app.app_context():
        for n, route in enumerate(routes):
            routes[n] = url_for(route)
            # yield url_for(route) #NOTE: This would be ideal, but not allowed.
            # convert the routes from func names to actual routes
    yield routes

@pytest.mark.parametrize('route', routes)
def test_page_load(client, route):
    assert client.get(route.endpoint).status_code == 200

I read up that you can't mix parametrize with a fixture as an argument due to something along the lines of interpretation/load/execution order, although, how is this solved in terms of 'best practice'?

I saw a solution where you can generate tests from a function directly, and that seems extremely flexible and might be along the lines of what I want Passing pytest fixture in parametrize (Although I can't use call a fixture decorated function directly, so probably not)

Although, I'm new to pytest and I'd love to see more examples of how to generate tests or perform multiple tests in an iteration with little-to-no restrictions while adhering to proper pytest styling and the DRY principle. (I know about conftest.py)

I'd prioritize versatility/practicality over proper styling if that matters. (within reason, maintainability is a high priority too)

I want to be able to reference the solution to this problem to help guide how I tackle structuring my tests in the future, but I seem to keep hitting roadblocks/limitations or being told by pytest I can't do X solution the way I would expect/want too.

Relevant Posts:

1. DRY: pytest: parameterize fixtures in a DRY way
2. Generate tests from a function: Passing pytest fixture in parametrize
3. Very simply solution (doesn't apply to this case): Parametrize pytest fixture
4. Flask app context in PyTest: Testing code that requires a Flask app or request context
5. Avoiding edge-cases with multiple list fixtures: Why does Pytest perform a nested loop over fixture parameters

Answer 1

My current solution that I fumbled my way across is this:

import pytest
from flask import url_for
from myflaskapp import get_app

@pytest.fixture
def app():
    app = get_app()
    # app context stuff trimmed out here
    return app

@pytest.fixture
def client(app):
    client = app.test_client()
    return client

def routes(app):
    '''GET method urls that we want to perform mass testing on'''
    routes = ['foo', 'bar']
    with app.app_context():
        for n, route in enumerate(routes):
            routes[n] = url_for(route)
    return routes

@pytest.mark.parametrize('route', routes(get_app()))
#NOTE: It'd be really nice if I could use routes as a 
# fixture and pytest would handle this for me. I feel like I'm
# breaking the rules here doing it this way. (But I don't think I actually am)
def test_page_load(client, route):
    assert client.get(route.endpoint).status_code == 200

My biggest issue with this solution is that, I can't call the fixture directly as a function, and this solution requires either that, or doing all the work my fixture does outside of the fixture, which is not ideal. I want to be able to reference this solution to tackle how I structure my tests in the future.

FOR ANYONE LOOKING TO COPY MY SOLUTION FOR FLASK SPECIFICALLY:

My current solution might be worse for some people than it is for me, I use a singleton structure for my get_app() so it should be fine if get_app() is called many times in my case, because it will call create_app() and store the app itself as a global variable if the global variable isn't already defined, basically emulating the behavior of only calling create_app() once.

Answer 2

Pytest fixtures themselves can be parameterized, though not with pytest.mark.parametrize. (It looks like this type of question was also answered here.) So:

import pytest
from flask import url_for
from myflaskapp import get_app

@pytest.fixture
def app():
    app = get_app()
    # app context stuff trimmed out here
    return app

@pytest.fixture
def client(app):
    client = app.test_client()
    return client

@pytest.fixture(params=[
    'foo', 
    'bar'
])
def route(request, app):
    '''GET method urls that we want to perform mass testing on'''
    with app.app_context():
        return url_for(request.param)


def test_page_load(client, route):
    assert client.get(route.endpoint).status_code == 200

The documentation explains it this way:

Fixture functions can be parametrized in which case they will be called multiple times, each time executing the set of dependent tests, i. e. the tests that depend on this fixture. Test functions usually do not need to be aware of their re-running. Fixture parametrization helps to write exhaustive functional tests for components which themselves can be configured in multiple ways.

Extending the previous example, we can flag the fixture to create two smtp_connection fixture instances which will cause all tests using the fixture to run twice. The fixture function gets access to each parameter through the special request object:

# content of conftest.py
import pytest import smtplib


@pytest.fixture(scope="module", params=["smtp.gmail.com", "mail.python.org"])
def smtp_connection(request):
    smtp_connection = smtplib.SMTP(request.param, 587, timeout=5)
    yield smtp_connection
    print("finalizing {}".format(smtp_connection))
    smtp_connection.close()

The main change is the declaration of params with @pytest.fixture, a list of values for each of which the fixture function will execute and can access a value via request.param. No test function code needs to change.