Importing modules from the project root

Question

I've seen lots of Python projects on GitHub which use really nice and clean imports, where they define their types in the root of the project, and import them from sub-packages within.

For example, you could have a directory structure like this:

project
    - foo
        - foo.py
    - bar
        - baz
            - baz.py
        - bar.py
    main.py
    types.py

There may be some class in types.py that is a core data type within the project.

# types.py

from dataclasses import dataclass


@dataclass(frozen=True)
class ImportantType:
    """This is a core type within the project that's used by sub-packages."""
    foo: str
    bar: str

Then, interestingly, within either sub-package you find code like this:

# foo/foo.py

from project import types

def example():
    f = types.ImportantType("foo", "bar")
    # does some processing on f.

From what I've learned, Python imports do not usually work like this. However, it seems really clean to be able to have your data models in the root of your project, and be able to import packages and modules relative to the root of the project.

How do you set up your project in order to be able to use imports like this?

An example of a project using this is thefuck, here's a link to the file importing from root.

Disclaimer: I have searched the code, and nowhere are they adding packages to PYTHONPATH, nor are they using the sys.path approach.

Answer 1

You only need to set PYTHONPATH when you do not install the package. Internally, Python searches sys.path to find whatever you specify with from or import keywords. The path stops at the root, those sub directories tells python that it needs to search a specific sequence of directories to find the correct file.

rootdir/
  project/
     __init__.py 
     foo/
        __init__.py # All __init__.py file can be empty
        hello_module.py
      tests/test_import.py
  setup.py

>>>> print(sys.path)
['', '/usr/local/Cellar/python@3.9/3.9.2_1/Frameworks/Python.framework/Versions/3.9/lib/python39.zip', '/usr/local/Cellar/python@3.9/3.9.2_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9', '/usr/local/Cellar/python@3.9/3.9.2_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/lib-dynload', '/usr/local/lib/python3.9/site-packages', '/usr/local/Cellar/protobuf/3.14.0/libexec/lib/python3.9/site-packages']

https://docs.python.org/3.5/library/sys.html#sys.path

#project/foo/hello_module.py
def example():
   return ("hello module")


#tests/test_import.py
import os, sys
import unittest

class TestStringMethods(unittest.TestCase):
    def test_import(self):
        try:
            from project.foo.hello_module import example
            self.assertEqual(example().lower(), 'hello module')
        except Exception:
            try:
                self.assertIsNone(os.environ["PYTHONPATH"], os.environ["PYTHONPATH"])
            except Exception:
                self.assertTrue(False, "Error PYTHONPATH not defined")
                pass
if __name__ == '__main__':
    unittest.main()

# setup.py
import setuptools
setuptools.setup(name='project',
  version='1.0',
  description='test import project',
  author='somebody not me',
  author_email='somebodynotme@example.net',
  packages=setuptools.find_packages(
    exclude=["test/"]
  )
 )

Test the project with PYTHONPATH

PYTHONPATH=`pwd` python3 tests/tests_import.py
.
----------------------------------------------------------------------
Ran 1 test in 0.004s

OK

Test the project without PYTHONPATH installed via virtualenv

## Setup venv
python3 -m venv venv
source venv/bin/activate

Notice how venv add an extra path

python3 -c "import sys; print(sys.path)"
['', '/usr/local/Cellar/python@3.9/3.9.2_1/Frameworks/Python.framework/Versions/3.9/lib/python39.zip', '/usr/local/Cellar/python@3.9/3.9.2_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9', '/usr/local/Cellar/python@3.9/3.9.2_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/lib-dynload', '/Users/somebodyotherthanme/stackoverflow/67140161/venv/lib/python3.9/site-packages']

Install test package and run it

pip install /path/to/rootdir
python3 tests/test_import.py

python3 tests/tests_import.py 
.
----------------------------------------------------------------------
Ran 1 test in 0.002s

OK

    (venv) ➜  root_dir/ ls venv/lib/python3.9/site-packages/project
__init__.py __pycache__ foo

Relative imports for the billionth time