How should I structure a pytest test only package?

Question

My use case it is about remotely (RESTful API, etc.) testing a sub-system of an overall system. This means the "pytest test only package" doesn't have any dependencies on production code (means other production code python packages).

I created a python package which just contains test related stuff like pytest tests, pytest fixtures, test helper functions in util modules, pytests conftest.py, pytests pytest.ini, etc. It doesn't contain any production code related stuff.

Right now the functionality is working but the structure of the package is quite "hacky". This means the installation is not working correctly (test, fixture and conftest files are not installed correctly into site_packages via MANIFEST.ini) and deployment of the package must be done "manually".

In the pytest docs I just found best practices about how to structure a package containing both, production and pytest test code: Tests outside application code, Tests as part of application code and Test directory structure.

1. How should I structure a python package which contains just test code? Are there alternative package structures (advantages, disadvantages)?
2. Where should test files and their dependencies (fixtures, helpers, etc.) be installed into?

Possible solutions for 2. : The avocado-framework deploys example tests as data files in setup.py. Dependent on the configuration the tests are deployed into /usr/share/avocado/tests per default.

Answer 1

Requirements

As I see it, you have three requirements to meet:

1. The installed tests should be easily discoverable
2. The tests should still be runnable locally for development purposes
3. Running the tests in both development mode and installed mode should produce identical outcome

Project structure

Considering that, here's how I would structure the project: create some dummy package (but with a unique distinguishable name) and put the tests dir in there, along with the conftest.py and the utilities modules. Here's how it would look like:

project/
├── setup.py
└── mypkg/
    ├── __init__.py
    └── tests/
        ├── conftest.py
        ├── utils.py
        ├── other_utils.py
        ├── test_spam.py
        ├── test_eggs.py
        └── other_tests/  # if you need to further grouping of tests
            ├── conftest.py  # if you need special conftest for other tests
            └── test_bacon.py

setup.py

from setuptools import setup

setup(
    name='mypkg-tests',
    version='0.1',
    install_requires=['pytest'],
    packages=['mypkg'],
    package_data={'mypkg': ['tests/*', 'tests/**/*']},
)

With this project structure:

1. Tests are correctly discovered and executed locally in the project/ dir:

   $ pytest -v
   ============================= test session starts =============================
   platform darwin -- Python 3.6.3, pytest-3.3.2, py-1.5.2, pluggy-0.6.0 -- /Users
   /hoefling/.virtualenvs/stackoverflow/bin/python
   cachedir: .cache
   rootdir: /Users/hoefling/projects/private/stackoverflow/so-48111426, inifile:
   plugins: forked-0.2, asyncio-0.8.0, xdist-1.22.0, mock-1.6.3, hypothesis-3.44.4
   collected 3 items
   
   spam/tests/test_eggs.py::test_foo PASSED                                 [ 33%]
   spam/tests/test_spam.py::test_bar PASSED                                 [ 66%]
   spam/tests/other_tests/test_bacon.py::test_baz PASSED                    [100%]
   
   ========================== 3 passed in 0.03 seconds ===========================
   

2. When you build a source tar or a wheel and install the package, it's easy to run the tests by providing the package name:

   $ pytest -pyargs mypkg
   ...
   

   You will get the exact same test result because the way pytest discovers the tests will be the same as when running them locally, only instead of scanning the current working directory, the package's directory will be scanned.

3. Although all the tests along with config and utils are installed in site-packages, they are not packages or modules themselves. To the outside world, the distribution contains only an empty package mypkg; nothing in tests/ is importable and will be only visible to pytest.

Why data_files is unreliable

Of course you could declare something like this in the setup script:

setup(
    ...
    data_files=[
        ('tests', ['tests/test_spam.py', 'tests/test_eggs.py']),  # etc
    ]
)

First of all, it's not that convenient to create and maintain a list of tests to be included (although it's surely possible to write some kind of custom tests lookup with os.walk or pathlib.glob or whatever). But more important is that you won't be able to reliably install the data_files to an absolute path. I want go into much details here; feel free to check out my other answer for more info - but basically, it's the wheel package that relativizes every absolute path in data_files to sys.prefix, and even if you build a source distribution, pip install will first build a wheel out of it, then install the wheel. So if you want to run the tests from installed package, you will first need to determine the sys.prefix and build the path yourself:

$ SYS_PREFIX=$(python -c "import sys; print(sys.prefix)")
$ pytest -v $SYS_PREFIX/whatever/dir/mapped/in/data_files/tests

The only way to avoid the wheel issue is to build the source distribution and install it with pip install mypkg --no-binary=mypkg. This option will force pip to skip the wheel building step and install directly from source. Only then the tests will be installed to an absolute path. I find it pretty inconvenient because there will be times when you forget the no-binary arg and will spend time looking for the error source. Or someone has to install the package while you aren't there to guide him, and won't be able to run the tests. Simply don't use data_files.