27

I have a module that sits in a namespace. Should tests and data the tests rely on go in the namespace or in the top level where setup.py sites?

./company/__init__.py
./company/namespace/__init__.py
./company/namespace/useful.py
./company/namespace/test_useful.py
./company/namespace/test_data/useful_data.xml
./setup.py

or

./company/__init__.py
./company/namespace/__init__.py
./company/namespace/useful.py
./test_useful.py
./test_data/useful_data.xml
./setup.py

Does the question amount to whether tests should be installed or not?

Stephen Paulger
  • 5,204
  • 3
  • 28
  • 46
  • 5
    possible duplicate of [Where do the Python unit tests go?](http://stackoverflow.com/questions/61151/where-do-the-python-unit-tests-go) – Wooble Mar 17 '11 at 15:16

3 Answers3

41

The Sample Project stores the tests outside the module.

The directory structure looks like this:

├── data
│   └── data_file
├── MANIFEST.in
├── README.rst
├── sample
│   ├── __init__.py
│   └── package_data.dat
├── setup.cfg
├── setup.py
└── tests
    ├── __init__.py
    └── test_simple.py

Related: The Packing Guide: https://packaging.python.org/en/latest/

Hint: Don't follow the "The Hitchhiker's Guide to Packaging". It has not been updated since 2010!

(do not confuse both pages. The "The Hitchhiker’s Guide to Python" is a very solid book)

guettli
  • 25,042
  • 81
  • 346
  • 663
  • 5
    See also https://www.jeffknupp.com/blog/2013/08/16/open-sourcing-a-python-project-the-right-way/, which puts the tests in the root project directory too. – jonrsharpe Oct 06 '15 at 07:42
  • 1
    Contradicting enough, this _The Hitchhiker's Guide to Python_ suggests placing the tests outside: http://docs.python-guide.org/en/latest/writing/structure/ – Kyr Jan 24 '17 at 11:33
  • 1
    @Kyr funny enough that is not what the article you linked says right now, I guess it got updated to suggest placing tests outside of main package – Maciej Urbański Mar 15 '19 at 14:49
  • @MaciejUrbański, thanks! Don't follow _The Hitchhiker's Guide to Python_ then! :) – Kyr Mar 15 '19 at 15:06
  • 1
    @Kyr do not confuse both pages. The "The Hitchhiker’s Guide to Python" is a very solid book – guettli Mar 15 '19 at 15:20
  • Given this example, how do you load the data_file in test_simple.py? I got a similar structure but run into problems when loading the file using open('../data/data_file'). It works if I run the test directly but fails when I run "python setup.py test" – nadre Jul 15 '19 at 07:40
  • I used this directory structure, and now I don't know how to distribute my package+tests correctly. Even when specifying `package-data` to include my tests directory `setup.py`... pip still installs `myModule`, then installs `tests` as a separate package. I have my `tests` directory just floating around in `Lib/site-packages` with no way to tell that it belongs to my module. How do you prevent this situation? – cowlinator Oct 30 '19 at 02:36
  • @cowlinator please create a new question. I think this will bring much better feedback. You can create a link here to your new question. – guettli Oct 30 '19 at 08:09
18

You should put your test module inside the module it tests according to The Hitchhiker's Guide to Packaging.

Here is their example:

TowelStuff/
    bin/
    CHANGES.txt
    docs/
    LICENSE.txt
    MANIFEST.in
    README.txt
    setup.py
    towelstuff/
        __init__.py
        location.py
        utils.py
        test/
            __init__.py
            test_location.py
            test_utils.py

This way your module will be distributed with its tests and users can use them to verify that it works with their set up.

See http://the-hitchhikers-guide-to-packaging.readthedocs.org/en/latest/creation.html.

HoldOffHunger
  • 18,769
  • 10
  • 104
  • 133
hwiechers
  • 14,583
  • 8
  • 53
  • 62
  • 6
    what's the import statement in the tests? – chiborg Jul 23 '14 at 09:37
  • 1
    Tests can use usual module imports. Tests often are running by absolute path and with customized PYTHONPATH environment variables, especially from IDE like PyCharm. – Denis Barmenkov Sep 23 '14 at 11:38
  • 2
    You shouldn't name the submodule test to avoid confusion with the standard library module also named test. – Ryan Jun 21 '15 at 23:02
  • 1
    Nevertheless, _The Hitchhiker's Guide to Python_ suggests placing the tests outside: docs.python-guide.org/en/latest/writing/structure! – Kyr Jan 24 '17 at 11:34
  • Ugh, should I trust *The Hitchhiker's Guide to Python*, or *The Hitchiker's Guide to Packaging*? – endolith Jun 13 '17 at 02:32
  • 4
    The "The Hitchhiker’s Guide to Packaging" was not updated since 2010. Please use this https://packaging.python.org/ guide – guettli Mar 15 '18 at 14:39
  • 1
    @guettli: I'm having a hard time finding anything concerning tests in that guide. Am I missing something? – Roman Reiner Apr 11 '19 at 08:11
  • @RomanReiner please tell this the authors of the guide. Please open an issue there. Thank you. – guettli Apr 11 '19 at 08:30
4

I personally create a single tests package as a sub package of the main package for a few reasons:

  • If tests is in parallel with the root package there's an off chance you, or a user may misconfigure setup.py and accidentally expose a global package named tests that will cause a great deal of confusion and headache until you realize what has happened. Putting it in the main module solves this as it's now under a (hopefully) globally unique namespace.

  • I don't like putting a test module within user package because test runners have to search through production code. This is probably not a problem for most. But, if you happen to be a hardware test engineer, you probably use the word 'test' a lot in your production code and don't want the unit test runner to pick that stuff up. It's much easier if all the tests are in one place separate from the production code.

  • I can further subdivide my tests folder into the types of tests, such as unit, functional and integration. My functional tests tend to have dependencies on weird proprietary hardware, data or are slow. So it's easy for me to continuously run just the fast unit test folder as I develop.

  • It can sometimes be convenient to have the tests be inside of the same package hierarchy as what it is testing.

Overall though, I think it's important to think for yourself about what's best for your particular problem domain after taking everyone's advice into account. 'Best practices' are great starting points, not end points, for developing a process.

willtalmadge
  • 406
  • 3
  • 7