pytest - test flow order

Question

I have a pytest code similar to below... if i run it with --count 3 it will run the test_first 3 times then test_second 3 times.

What if i would like it to run test_first, test_second and repeat that flow?

Thanks.:)

@pytest.mark.usefixtures('setup')
class TestSomething:

    def run_setup(self):
        pass

    def test_first(self):
        print('test 1')
        name = 'name'
        assert name.isalpha()

    def test_second(self):
        print('test 2')
        name = '12345'
        assert name.isalpha()

Like it much better when people put all the code for a fully executable example. — Justin Malinchak, Jul 26 '18 at 13:12
I dont think the whole code is relevant, it seems like a specific pytest behaviour , correct me if i'm wrong please :) — Isaac A, Jul 26 '18 at 13:15
for what it worth i have the conftest.py file which have `@pytest.fixture() def setup(): print('Setup')` — Isaac A, Jul 26 '18 at 13:17
A naive question, py.test calls only methods starting with test_, so why should it call first_test and second_test at all? (Please educate me if I am wrong!) Also, couldn't you simply add a third method that calls the two methods? — physicsGuy, Jul 26 '18 at 13:25
@physicsGuy You are right! i'm correcting the example but the question remains. (consider it as test_first, test_second). as for the second part, third method that calls the two methods would be considered as 1 test. while in reality those are 2 tests dependent on each other. — Isaac A, Jul 26 '18 at 13:32
@Evgeny no, you can ignore this one. Also dependency plugin is no good for me here. Tried it. All is needed is for the test to run multiple times in synced order. For ex; 1,2,3 1,2,3 etc... — Isaac A, Jul 26 '18 at 15:11
For flow testing, look at `hypothesis` and specifically what it can do about state machines: https://hypothesis.readthedocs.io/en/latest/stateful.html and you can find fun examples using it like https://hypothesis.works/articles/how-not-to-die-hard-with-hypothesis/ — Patrick Mevzek, May 20 '21 at 23:22

score 2 · Answer 1 · answered Jul 26 '18 at 14:45

2

You can implement it yourself. Take a look at the pytest_collection_modifyitems hook where you can alter the list of tests to be executed. Example:

# conftest.py
import pytest

def pytest_addoption(parser):
    parser.addoption('--numrepeats', action='store', type=int, default=1)

def pytest_collection_modifyitems(items):
    numrepeats = pytest.config.getoption('--numrepeats')
    items.extend(items * (numrepeats - 1))

When put into a conftest.py file in the tests root dir, this code adds a new command line option numrepeats that will repeat the test run n times:

$ pytest --numrepeats 3

answered Jul 26 '18 at 14:45

hoefling

59,418
12
147
194

Unfortunately this only works on global scope, thus it is not possible to do the repeat of just tests in one module. – Kjeld Flarup Jul 31 '18 at 11:42
@KjeldFlarup filter the items by module and append them at the end of the items list. – hoefling Jul 31 '18 at 12:09
@KjeldFlarup you can run `pytest` only on the module of interest or use `-k` to select a subset of tests to be run. – Stefano Messina Oct 02 '18 at 09:39

score 1 · Answer 2 · answered Jul 26 '18 at 14:03

1

Based on https://pytest-ordering.readthedocs.io (alpha) plug-in you can do:

import pytest

@pytest.mark.order2
def test_foo():
    assert True

@pytest.mark.order1
def test_bar():
    assert True

See also a discussion at Test case execution order in pytest.

My personal take on this if your tests require sequence, they are not really well isolated and some other test suit design is possible.

answered Jul 26 '18 at 14:03

Evgeny

4,173
2
19
39

well, this works only if i dont use --count arg. once i want to run the tests multiple times it will run first test x times then second test x times.... – Isaac A Jul 26 '18 at 14:08

pytest - test flow order

2 Answers2

Linked