Can I use a python module's main for testing?

Question

I'm developing a Python library on Python 3.8.2 and I would like to run a module as main for testing purposes. When I try, I get a ModuleNotFound error.

Here's my library structure:

.
├── foo
│   ├── __init__.py
│   ├── bar.py
│   └── quux
│       ├── __init__.py
│       ├── corge.py
│       └── garply.py
├── main.py

bary.py:

def baz():
    print("baz")

corge.py

from foo.quux.garply import *


def grault():
    waldo()
    print("grault")


if __name__ == '__main__':
    grault()

garply.py

def waldo():
    print("waldo")

main.py

from foo.bar import *
from foo.quux.corge import *

if __name__ == '__main__':
    baz()
    grault()

(All __init__.py files are empty)

When I run main.py, it works.

$ python main.py
baz
waldo
grault

If I try to run corge.py, I get the following error:

$ python foo/quux/corge.py 
Traceback (most recent call last):
  File "foo/quux/corge.py", line 1, in <module>
    from foo.quux.garply import *
ModuleNotFoundError: No module named 'foo'

It doesn't matter what my current working directory is, it always gives this error.

$ cd foo/quux/
$ python corge.py 
Traceback (most recent call last):
  File "corge.py", line 1, in <module>
    from foo.quux.garply import *
ModuleNotFoundError: No module named 'foo'

While I was testing this, I created a new PyCharm project with PyCharm 2020.1 and implemented the structure I described. To my surprise, it works with with the default detected run configuration.

I tried using the venv that PyCharm automatically creates, but it still did not work. It does not work if I directly copy/paste the command and use their CWD. It does not work with the terminal built into PyCharm. It only works with the PyCharm Run button.

Am I doing something wrong with my module structure? If so, what could PyCharm be doing to make this work? If not, why does it not work outside of PyCharm?

Does this answer your question? [Python ModuleNotFoundError Testsuite (Import Error)](https://stackoverflow.com/questions/48329498/python-modulenotfounderror-testsuite-import-error) — Joe, Apr 16 '20 at 16:31
https://stackoverflow.com/search?q=python+modulenotfounderror — Joe, Apr 16 '20 at 16:31

score 1 · Answer 1 · answered Apr 16 '20 at 16:13

Your invocation of main.py works because Python will look in immediate sub-directories for modules. Any module in any other location that wants to import foo.yadda.whatever will have to find foo by searching your PYTHONPATH. Therefore you need to add the parent directory of foo to your PYTHONPATH.

score 1 · Accepted Answer · answered Apr 16 '20 at 16:31

Python needs to know the directory where foo is located in order to import it. sys.path lists the directories where python will search.

When you install a package, the installer worries about doing that - usually by placing the module in a well-known directory or adding the install package path to sys.path.

When you run a script, python automatically adds that script's path to sys.path, so when you run main.py, you will find foo.

How to run as module as __main__

One option is to make a package installable (setup.py, wheels, etc....) and use develop mode ("pip install --editable ./" vs "python setup.py develop" for some discussion). That's what I do when I am developing something I plan on making available to others.

Another is to add your directory to PYTHONPATH, perhaps even as you run the program. On linux that would be

PYTHONPATH=path/to/fooproject:$PYTHONPATH python foo/quux/corge.py

Yet another, and I do this one too, is to hack the path in the module itself. __file__ gives your filename relative to current working directory and you know how deep you are in your package hierarchy. So you can just make __file__ absolute and peel off a couple of directory names

corge.py

import sys
import os

if __name__ == "__main__":
    # I'm two levels deep in the package so package directory is
    packagedir = os.path.abspath(os.path.join(os.path.dirname(__file__),
        "..", ".."))
    sys.path.insert(0, packagedir)
    import foo

Finally, don't do it in the first place. When you run corge.py as a script, it gets a different namespace __main__ than foo.bar.corge imported as a module. Its global variables / classes / functions are loaded twice and you get different ones depending on whether you call them through the __main__ namespace or foo.bar.corge.

Its better to take anything you are tempted to put into a main in corge.py, make them separate scripts. For instance, you could add def main() to your modules. In main.py you could add an option --run foo.bar.corge telling main to import corge.py and run its main(). argparse subcommands can be used for this.

That makes a lot of sense, thank you! My original plan was to use this to run unit tests, would it be appropriate to define a `test()` method in the module, or should that be handled differently? — TechnoSam, Apr 16 '20 at 17:50
Yes, you can put the tests with the code - I'm not sure that its common practice but it wouldn't raise an eyebrow if you did. But you are putting code in a module that doesn't need to be there in normal runtime. I've tended to use `nose` or just the vanilla unittest that comes with python (pytest is another option) and I separate them into a separate directory. I think you should do whatever is the most normal thing for your test framework. — tdelaney, Apr 16 '20 at 22:13

Can I use a python module's main for testing?

2 Answers2