Interleaved repeat of pytest

Question

I have some tests which I would like to repeat a number of times. I tried the pytest-repeat plugin

pip3 install pytest-repeat

import pytest
@pytest.mark.repeat(2)
class TestDemo():
    def test_demo1(self):
        pass
    def test_demo2(self):
        pass

This works

test_class_repeat.py::TestDemo::test_demo1[1/2] PASSED
test_class_repeat.py::TestDemo::test_demo1[2/2] PASSED
test_class_repeat.py::TestDemo::test_demo2[1/2] PASSED
test_class_repeat.py::TestDemo::test_demo2[2/2] PASSED

Except that I want an interleaved order running all tests, and the run all tests again

test_class_repeat.py::TestDemo::test_demo1[1/2] PASSED
test_class_repeat.py::TestDemo::test_demo2[1/2] PASSED
test_class_repeat.py::TestDemo::test_demo1[2/2] PASSED
test_class_repeat.py::TestDemo::test_demo2[2/2] PASSED

Is there a simple way to do this?

Possible duplicate of [pytest - test flow order](https://stackoverflow.com/questions/51539570/pytest-test-flow-order) — hoefling, Jul 31 '18 at 10:54

score 0 · Answer 1 · answered Oct 02 '18 at 10:22

Well, not a very clean solution, naively just define a test function that executes the interleaved tests while skipping the definition itself, and apply repeat on that:

import pytest

@pytest.mark.skip(reason='Definition only')
class TestDemo():
    def test_demo1(self):
        print('In Test 1')
        assert 1 == 1
    def test_demo2(self):
        print('In Test 2')
        assert 2 == 2


@pytest.mark.repeat(2)
def test_all():
    demo = TestDemo()
    demo.test_demo1()
    demo.test_demo2()

Execution (in jupyter notebook) gives:

Test.py::TestDemo::test_demo1 SKIPPED
Test.py::TestDemo::test_demo2 SKIPPED
Test.py::test_all[1/2] 
In Test 1
In Test 2
PASSED
TestProject.py::test_all[2/2] 
In Test 1
In Test 2
PASSED

Side note: if one of the two nested tests does not pass test_all does not pass, something desired from interleaved tests?

This unfortunately does not report test_demo1 and test_demo2 individually. — Kjeld Flarup, Oct 04 '18 at 08:23

score 0 · Answer 2 · answered Oct 19 '18 at 02:04

0

You can use pytest-flakefinder package by dropbox. It repeats tests after the run is complete.

Usage: py.test --flake-finder --flake-runs=runs.

answered Oct 19 '18 at 02:04

SilentGuy

1,867
17
27

score 0 · Answer 3 · answered Jul 30 '19 at 15:18

This can be done using the pytest.mark.parametrize if the functions have a parameter. Below is an example.

import pytest
iter_list = [1,2,3]
@pytest.mark.parametrize('param1', iter_list, scope = 'class')
class TestDemo():
    def test_demo1(self, param1):
        pass
    def test_demo2(self, param1):
        pass

Interleaved repeat of pytest

3 Answers3