-1

Following this answer and the doc, I am trying to make tests depend on each other between files:

test_connectivity.py:

@pytest.mark.dependency(depends=["test_services_up.py::test_sssss"], scope="session")
def test_aaa():
    assert False


@pytest.mark.dependency(depends=["tests/test_services_up.py::test_sssss"], scope="session")
def test_bbb():
    assert False

@pytest.mark.dependency(depends=["atp/tests/test_services_up.py::test_sssss"], scope="session")
def test_ccc():
    assert False

and test_services_up.py:

@pytest.mark.dependency()
def test_sssss():
    assert True

The folder structure:

    atp
    ----|tests
    --------|test_connectivity.py
    --------|test_services_up.py

Output:

> ssh://me@host:port~/src/uv-fleet-atp/venv/bin/python -u ~/.pycharm_helpers/pycharm/_jb_pytest_runner.py --path ~/src/uv-fleet-atp/atp/tests
============================= test session starts ==============================
collected 4 items                                                              

test_connectivity.py::test_aaa SKIPPED (test_aaa depends on test_ser...)
Skipped: test_aaa depends on test_services_up.py::test_sssss

test_connectivity.py::test_bbb SKIPPED (test_bbb depends on tests/te...)
Skipped: test_bbb depends on tests/test_services_up.py::test_sssss

test_connectivity.py::test_ccc SKIPPED (test_ccc depends on atp/test...)
Skipped: test_ccc depends on atp/tests/test_services_up.py::test_sssss

test_services_up.py::test_sssss PASSED


========================= 1 passed, 3 skipped in 0.35s =========================

How to not skip dependent tests?

Gulzar
  • 23,452
  • 27
  • 113
  • 201
  • The problem is the order of the tests. `pytest-dependency` does not order the tests, but if you install the `pytest-order` plugin and use the `order-dependencies` flag, the tests will be ordered so that tests that other tests depend on are run first. – MrBean Bremen Feb 13 '22 at 09:53
  • @MrBeanBremen Please explain. What's the point of dependency without order? This doesn't really make sense. Can you add code to show what you mean please? – Gulzar Feb 13 '22 at 10:02
  • Also, clearly at least 2 of these tests don't exist, yet there isn't even a warning – Gulzar Feb 13 '22 at 10:03

2 Answers2

1

As mentioned in the comments, pytest-dependency does not order tests, it relies on the tests executed in the correct order for dependencies to work. The reasoning behind this is that pytest-dependency is thought to do one thing and one thing only, which is to skip tests depending on the test relationships. Ordering can either be done manually by arranging the tests accordingly (e.g. adapt the names, what most users seem to do), or rely on an ordering plugin. There has been a controversial discussion about this, and certainly not everyone agrees with that philosophy (I tend to agree), but in the end it is the prerogative of the plugin author to decide.

pytest-order works together with pytest-dependency. If it is installed and you run your tests with the option --order-dependencies it will order your tests with dependency markers if needed. As usual, you can instead add the option to pytest.ini:

[pytest]
; always order tests with dependency markers
addopts = --order-dependencies

As for tests that cannot be found, it should issue a respective warning.

Disclaimer:
I'm the maintainer of pytest-order.

MrBean Bremen
  • 14,916
  • 3
  • 26
  • 46
  • Thanks! So I did `pip install pytest-order` and `pytest_plugins = ["dependency", "order"]`. Getting `ImportError: Error importing plugin "order": No module named 'order'` – Gulzar Feb 13 '22 at 11:30
  • No need to add this to `pytest-plugins`, though I will have a look at this - probably using `pytest-order` instead of `order` will work. – MrBean Bremen Feb 13 '22 at 11:34
  • Without adding, it works. With adding as `pytest-order`, still doesn't. Thanks! – Gulzar Feb 13 '22 at 11:37
  • Also, how do people order their tests without the plugin? alphabetically is so error prone... – Gulzar Feb 13 '22 at 11:38
  • Ok, it should read `pytest_order` instead, but that is obviously not documented... After a quick survey of other pytest plugins it turns out that most seem to use this (`pytest_xxx`) as entry point, so I'll probably keep it, but add it to the documentation. – MrBean Bremen Feb 13 '22 at 12:03
  • Added a pytest-order SO tag you may want to watch – Gulzar Feb 13 '22 at 13:04
  • Does ordering also work with xdist to prevent race conditions? – Gulzar Feb 13 '22 at 13:33
  • Not really, see the [documentation](https://pytest-dev.github.io/pytest-order/stable/other_plugins.html#usage-with-pytest-xdist) – MrBean Bremen Feb 13 '22 at 13:43
  • I am trying to run a single test with this usage, but if it depends on another test, it just doesn't run, instead of first calling that test. Can this be worked around? I don't want to have to run the entire suite every time – Gulzar Feb 15 '22 at 17:12
  • 1
    I don't know of a feature or plugin that does that, and it is probably not trivial to write one, as it would involve calling a test that is not loaded yet. Sorry, no idea at the moment... And thanks for the tag, BTW - I'm watching it now. – MrBean Bremen Feb 15 '22 at 18:08
  • Why does it require to calling a not loaded test? Don't all tests get loaded, then some get skipped by static information? – Gulzar Feb 15 '22 at 20:10
  • I meant loaded in the sense of processed by pytest, so that it will be available in the pytest hooks. You may ask a separate question regarding this, there are certainly more knowledgeable folks out there. – MrBean Bremen Feb 15 '22 at 20:31
1

I had the same issue, but after while I found pytest-depends plugin, which is close similar to the dependency plugin but the main difference is 'depends' works as you want. So If you have all tests in one direction but the dependency is only on test files then you can try with:

@pytest.mark.depends(on=["test_services_up.py::test_sssss"], scope="session")
def test_aaa():
    assert False


@pytest.mark.depends(on=["test_services_up.py::test_sssss"], scope="session")
def test_bbb():
    assert False

@pytest.mark.depends(on=["test_services_up.py::test_sssss"], scope="session")
def test_ccc():
    assert False


test_services_up.py
@pytest.mark.depends()
def test_sssss():
    assert True

This should help and work, 'depends' plugin has also auto order, which means that it will run firstly main tests. You can find more here: https://pypi.org/project/pytest-depends/

Matelko
  • 15
  • 1
  • 5