5

TL;DR: If I'm using pytest and some other test-only dependencies, is it possible to assert that none of these test-only dependencies have leaked into actual, non-test code?

In Java, when I run tests, the tests themselves and the code-under-test will run in different classloaders, with their own dependencies scoped there. So if the code-under-test inadvertently referenced testng, the tests will fail, even though testng is running the tests.

Can the same be achieved in Python? If my non-test code accidentally imports pytest, can I catch that and have the tests fail? I can't see how to achieve that.

Although setuptools and pip etc. make it relatively easy to keep install/run and dev/test dependencies separate, and even avoid polluting the current venv, they are still present when the tests run. This means that running python setup.py test for a module can pass, but python setup.py install followed by something as simple as importing the module can instafail.

Given: cat setup.py

from setuptools import setup, find_packages

setup(
    name="brettpy",
    version="0.0.1",
    packages=find_packages(),
    setup_requires=["pytest-runner",],
    tests_require=["pytest",],
)

cat setup.cfg

[aliases]
test=pytest

cat brettpy/__init__.py

import pytest

cat tests/test_brettpy.py

def test_import_brettpy():
    import brettpy
    del brettpy

def test_run_main():
    from brettpy import __main__ as main
    main.main()

... python setup.py test will pass, but python setup.py install && python -m brettpy will fail with:

ModuleNotFoundError: No module named 'pytest'

How do others ensure test-dependencies don't bleed into real code and cause missing-dependencies bugs when installed? Feels like a bug that the test-framework should be able to catch.

javabrett
  • 7,020
  • 4
  • 51
  • 73
  • I don’t think it’s possible. You are running the tests on a certain virtual environment which contains all the modules. The only way to know whether a module is a test dependency or not, is to check the lists located in setup.py. This is a nice idea for an open source project though. – AdamGold Jun 07 '19 at 01:15
  • This article https://realpython.com/python-testing/#how-to-write-assertions lists how to use tox for different environment (I'm still not sure if you need pytest or unittest in these environments). I will check out and come back. If you don't need pytest or unittest in these environments, your requirement is satisfied. – Edward Aung Jun 07 '19 at 01:17
  • Usually an IDE (I know pycharm does) will have the ability to highlight non-valid import statements for you. Is that an option? – Arne Jun 09 '19 at 14:24
  • The IDE will probably recognize it as a valid import since it's running on an environment which includes the test modules. – AdamGold Jun 12 '19 at 06:53
  • 1
    I don't think an IDE is suitable here, although it would surely help if it could easily detect bad imports - the goal here is for this to be something automated-testing would pick-up. – javabrett Jun 12 '19 at 09:17

2 Answers2

3

You can explicitly test for test imports by using Python's modulefinder. This is possibly not the prettiest of solutions:

from modulefinder import ModuleFinder

def test_imports():
    finder = ModuleFinder()
    finder.run_script("PATH_TO_MAIN_SCRIPT")
    tests_require = ["pytest",]  # you can also get this list straight from `setup.py`
    overlapping = [mod for mod in finder.modules.keys() for req in tests_require if req in mod]
    assert not overlapping

Or if you don't want to use a list comprehension:

    for mod_name in finder.modules.keys():
        for req in tests_require:
            assert req not in mod_name  # module could be pytest.something, _pytest, etc.
AdamGold
  • 4,941
  • 4
  • 29
  • 47
  • 1
    This is an excellent answer and works, bounty awarded. I do suspect there will be improvements though. My main concern is that this depends on reachability of the problem import from the chosen entry-point. For an application, I think `__main__` would need to be called in a way that will find every bad import, which seems unlikely. Libraries might not even have a `main`. So a better solution if possible would instrument existing test-execution, so existing coverage is enhanced with the no-test-deps-import-checking. – javabrett Jun 12 '19 at 09:33
  • I agree. A perfect solution would be to search through all the files, just a like a formatter does, and compare with a list of bad imports. Shouldn't be too hard to achieve, though I think that such a thing should be a separate module and not an explicit test. – AdamGold Jun 12 '19 at 10:22
-4

You can try something like this: https://stackoverflow.com/a/1592578/9095840

def no_test():
    try:
        pytest
    except NameError:
        pass
    else:
        exit()
markemus
  • 1,702
  • 15
  • 23
  • 2
    Could you expand on your answer? I can't see how this solves the problem. `pytest` will be installed, since we are running tests. Is this code-snippet a test - how would it test all other non-test-code to show it doesn't `import pytest`? This seems to test itself. – javabrett Jun 07 '19 at 01:16
  • It checks whether the variable "pytest" is defined. If it is, it calls the exit() function (or you can do something else if you prefer). You would have to call it in each module though. – markemus Jun 07 '19 at 01:19
  • Actually here's a much better solution that actually checks whether the module has been imported anywhere or not: https://stackoverflow.com/a/30483269/9095840 – markemus Jun 07 '19 at 01:21