pytest: Mark a test as a "must pass" and stop testing if it fails

Question

I am trying to implement a new pytest marker, called @pytest.mark.must_pass, to indicate that if the marked test fails, pytest should skip all subsequent tests and terminate.

I have been able to use the pytest_runtest_call hook to get pytest to terminate if the marked test failed, but I am using pytest.exit, which does not print a traceback, nor does it show the failure indication for the test in question.

I need this failure to appear as any other test failure, except that pytest stops testing after it prints whatever it needs to print to detail the failure.

My code so far:

# Copied this implementation from _pytest.runner
def pytest_runtest_call(item):
    _update_current_test_var(item, "call")
    try:
        del sys.last_type
        del sys.last_value
        del sys.last_traceback
    except AttributeError:
        pass
    try:
        item.runtest()
    except Exception:
        # Store trace info to allow postmortem debugging
        type, value, tb = sys.exc_info()
        assert tb is not None
        tb = tb.tb_next  # Skip *this* frame
        sys.last_type = type
        sys.last_value = value
        sys.last_traceback = tb
        del type, value, tb  # Get rid of these in this frame

        # If test is marked as must pass, stop testing here
        if item.iter_markers(name = "must_pass"):
            pytest.exit('Test marked as "must_pass" failed, terminating.')

        raise

Is there already a mechanism for doing this built into pytest?

Any help will be greatly appreciated.

Thanks.

based on the code you provided, pytest will always exit once you get to the `must_pass` marker — gold_cy, Dec 17 '19 at 17:40
@aws_apprentice Yes, but it will not show up as a failed test. It will not show the traceback, or print "FAILED" in red, for instance. — Yoel Einhoren, Dec 17 '19 at 18:04
so what exactly is the desired behavior, if it encounter this mark to not run any more tests and just stop? — gold_cy, Dec 17 '19 at 18:22
Yes, when a test marked as "must pass" fails, it should display the failure as any other test failure, and skip all the remaining tests, then exit. — Yoel Einhoren, Dec 18 '19 at 08:19

score 3 · Accepted Answer · answered Dec 18 '19 at 12:43

So this can be achieved by using pytest_runtest_makereport and pytest_runtest_setup

In your conftest.py you would place the following:

import pytest


def pytest_runtest_makereport(item, call):
    if item.iter_markers(name='must_pass'):
        if call.excinfo is not None:
            parent = item.parent
            parent._mpfailed = item


def pytest_runtest_setup(item):
    must_pass_failed = getattr(item.parent, '_mpfailed', None)
    if must_pass_failed is not None:
        pytest.skip('must pass test failed (%s)' % must_pass_failed.name)

And now when we test it with the following:

import pytest


def foo(a, b):
    return a + b


def test_foo_1():
    assert foo(1, 1) == 2


@pytest.mark.must_pass
def test_foo_2():
    assert foo(2, 2) == 6


def test_foo_3():
    assert foo(3, 3) == 6


def test_foo_4():
    assert foo(4, 4) == 8

We see the desired output:

     ▲ = pytest test.py 
=============================================================== test session starts ================================================================
platform darwin -- Python 3.6.5, pytest-4.6.2, py-1.8.0, pluggy-0.12.0
rootdir: /Users/foo/Desktop/testing, inifile: pytest.ini
plugins: cov-2.7.1
collected 4 items                                                                                                                                  

test.py .Fss                                                                                                                                 [100%]

===================================================================== FAILURES =====================================================================
____________________________________________________________________ test_foo_2 ____________________________________________________________________

    @pytest.mark.must_pass
    def test_foo_2():
>       assert foo(2, 2) == 6
E       assert 4 == 6
E        +  where 4 = foo(2, 2)

test.py:14: AssertionError
================================================== 1 failed, 1 passed, 2 skipped in 0.08 seconds ===================================================

How to use this same test pass if I have multiple files with multiple tests? — VikasKM, Apr 20 '22 at 07:36
@VikasKM that would be up to you to figure out, the answer here was not directed at such a question. you have the basis here to build a solution that fits your need. — gold_cy, Apr 20 '22 at 13:22
I would add "if item.iter_markers(name='skip'): return" in pytest_runtest_makereport so it doesn't expect a skipped test to pass. — Jake OPJ, Aug 10 '23 at 18:27

score 0 · Answer 2 · answered Mar 15 '21 at 15:54

0

Pasting the code in the accepted answer yields this:

'must_pass' not found in `markers` configuration option

For anyone coming here wanting to use – not implement – this, the same can be achieved with pytest-dependency:

pip install pytest-dependency
See this answer for usage.

answered Mar 15 '21 at 15:54

user2394284

5,520
4
32
38

2

as noted, adding a 3rd party library for this is not needed, simply register the marker in your `pytest.ini` as shown [here](https://docs.pytest.org/en/stable/example/markers.html#registering-markers) – gold_cy Mar 15 '21 at 18:16

pytest: Mark a test as a "must pass" and stop testing if it fails

2 Answers2

Linked