pytest running scenarios in the correct order in the class

Question

So I have the following structure:

class Test(object):

   def test_1(self):
      pass

   def test_2(self):
      pass

   def test_3(self):
      pass

it runs great, NOW I'm adding the "scenarios" (as it's recommended at pytest - A quick port of “testscenarios”):

def pytest_generate_tests(metafunc):
    idlist = []
    argvalues = []
    for scenario in metafunc.cls.scenarios:
        idlist.append(scenario[0])
        items = scenario[1].items()
        argnames = [x[0] for x in items]
        argvalues.append(([x[1] for x in items]))
    metafunc.parametrize(argnames, argvalues, ids=idlist)

class Test(object):
       scenarios = ['1' {'arg':'value1'},
                    '2' {'arg':'value2'}]

       def test_1(self, arg):
          pass

       def test_2(self, arg):
          pass

       def test_3(self, arg):
          pass

When I run it the ORDER of tests is wrong, I get:

test_1[1]  
test_1[2]  
test_2[1]   
test_2[2]  
test_3[1]  
test_3[2]

Doesn't really look like a scenario for the Test class.

QUESTION: Is the any solution to run it in the correct order? like:

test_1[1]
test_2[1]
test_3[1]
test_1[2]
test_2[2]
test_3[2]

score 5 · Accepted Answer · edited Aug 10 '17 at 04:28

The upcoming pytest-2.3 has support for better (resource-based) ordering, and i just updated the scenario example in the docs: https://docs.pytest.org/en/latest/example/parametrize.html#a-quick-port-of-testscenarios

You can preliminary install the current development version with

pip install -i http://pypi.testrun.org -U pytest

and should get pytest-2.3.0.dev15 with "py.test --version" and be able to use it.

score 1 · Answer 2 · answered Sep 21 '12 at 03:12

1

py.test runs tests in a distributed fashion, which means the order is essentially random.

You should use the -n option and set the process number to 1. Then tests should be run in alphabetical order by the single process spawned.

More than this I don't know if you can do. Anyway depending on the order of tests is generally bad design. So you should try to not depend on it at all.

answered Sep 21 '12 at 03:12

Bakuriu

98,325
22
197
231

I think Alex was not referring to distributed testing here, otherwise i mostly agree. – hpk42 Sep 21 '12 at 06:36
@Bakuriu Thanks for your answer, however indeed I was referring to the order of tests within the same process. Also about the order dependency - in my opinion, it's very arguable: test Classes should be independent, however test methods within - are my test steps. I'm using pytest not for unit tests, but for UI automation with Selenium webdriver. – Alex Okrushko Sep 21 '12 at 12:24
Latest py.test does not seem to support -n, any ideas? – Oliver Dec 22 '15 at 23:12

pytest running scenarios in the correct order in the class

2 Answers2

Linked