How to import without using sys.path?

Question

I have a python3 project organized like this:

project
    src
        __init.py__
        file1.py (contains Class1)
        file2.py (contains Class2)
    test
        test1.py (contains TestClass1)
        test2.py (contains TestClass2)

I want to be able to import classes from file1.py into file2.py and from either file1.py or file2.py into test1.py and test2.py. What's the most straightforward way to accomplish this without setting sys.path?

I'm using python 3.7.

The code gets called as:

cd project/src/ && python3 file2.py

and as:

cd project/ && python3 -m unittest discover

This is an open-source project, so anyone who wants is welcome to the URL for our GitHub repo, but I expect it's not relevant to the question.

This ought not to be difficult but it's giving me fits. I've read the Python3 import chapter and searched for canonical Python project organization, to no avail.

Is there a particular reason you don't want to use `sys.path`? Could use set the `PYTHONPATH` environment variable before running your tests? — kingkupps, Jun 29 '20 at 03:21
That's a reasonable question. To me, sys.path seems like a hack. I find it difficult to believe there's not a more straightforward way to accomplish things in a simple project. Other languages, both older and newer than Python, have no difficulty with this. — Adam Wildavsky, Jun 29 '20 at 04:55
This project structure looks weird. A setup like this means the package itself is called ``src`` – usually, ``src`` is a normal directory which *contains* the package. Why is there a ``src/__init__.py`` when ``src/file2.py`` gets executed directly? A package implies that ``file2.py`` gets executed as part of the package, e.g. ``python3 -m package.src.file2``. Which parts *are* the package and which ones are just directories? — MisterMiyagi, Jun 30 '20 at 19:18
Does this answer your question? [Import a module from both within same package and from outside the package in Python 3](https://stackoverflow.com/questions/47319423/import-a-module-from-both-within-same-package-and-from-outside-the-package-in-py) — MisterMiyagi, Jun 30 '20 at 19:30
Your link helps a lot, @MisterMiyagi - thanks! The "-m" flag is indeed the secret sauce. I now have something working. I'll play to document what I've got in an answer to my own question. — Adam Wildavsky, Jul 04 '20 at 04:14
@MisterMiyagi, I do not understand what it is about my project structure that looks weird. How would you structure a simple project? I've looked around for canonical Python directory structures but did not find much. — Adam Wildavsky, Jul 04 '20 at 04:16
@AdamWildavsky The weird part is that your *package* appears to be named ``src``. Usually, ``src`` is a regular folder in which the actual package is. So instead of ``project/src/__init__.py``, one would have ``project/src/my_package/__init__.py``. Naming the package ``src`` is not wrong, just weird. Package layout and the ``src`` directory are [discussed in this blog post](https://blog.ionelmc.ro/2014/05/25/python-packaging/#the-structure), for example. — MisterMiyagi, Jul 07 '20 at 08:06
I see now! Yes, it would seem weird if someone outside our project were to import our package, and I can't say that will never happen. Thanks again, @MisterMiyagi. — Adam Wildavsky, Jul 10 '20 at 03:49

kingkupps · Answer 1 · 2020-06-30T19:04:07.030

0

Using `init.py`

Python uses the presence of __init__.py files to determine which directories are Python packages. If you add these in your project like so:

project
    __init__.py
    src
        __init__.py
        file1.py (contains Class1)
        file2.py (contains Class2)
    test
        __init__.py
        test1.py (contains TestClass1)
        test2.py (contains TestClass2)

You should be able to import your source from your tests via the import path: from project.src.file1 import Class1.

Then run your tests with: python -m unittest discover -s /path/to/project/test

Using `setuptools`

You could use setuptools to install your source then run something like this:

cd project/ && python3 -m pip install . && python3 -m unittest discover

You just need to add a setup.py file to the root of your project. For a simple setup.py script, you might want to refer to https://github.com/google/mobly/blob/master/setup.py (a project I used to use quite a bit). There are plenty of projects that use setuptools but many of them do pretty complicated setup and it can often be difficult to see exactly what's going on.

edited Jun 30 '20 at 19:04

answered Jun 29 '20 at 06:49

kingkupps

3,284
2
16
28

Thanks for the suggestion. I was not aware of setuptools. It seems like overkill for such a simple project. Appending to sys.path is less work, and that's still something I hope to avoid! – Adam Wildavsky Jun 30 '20 at 04:12
No prob! Unfortunately, outside of these three solutions, I don't think you'll find a simpler way to import python code outside of a given package tree. You could also consider putting an `__init__.py` in the root of your repo and naming your directories accordingly but that might require a larger refactor. – kingkupps Jun 30 '20 at 05:02
Thanks, @kingkupps. I'd be happy to add an __init__.py in the root directory if that would help! Can you give me a hint as to the import syntax I'd use if I do that? – Adam Wildavsky Jun 30 '20 at 15:59
Just updated my answer! You'll probably want to rename some of your project tree if you go with that route – kingkupps Jun 30 '20 at 16:57
Thanks! A friend who's not on StackOverflow suggested doing similarly. I'll try it out and report back. – Adam Wildavsky Jul 02 '20 at 19:14
The **Using __init__.py** suggestion doesn't seem to work. I get this error when executing `python3 file2.py`: `ModuleNotFoundError: No module named 'project'` I did find an alternative, though - I'll post again tomorrow (Saturday.) – Adam Wildavsky Jul 03 '20 at 06:19
1

Aha! It will work if I invoke is as `cd project/.. && python3 -m project.src.file2` My alternative requires `python3 -m` as well. – Adam Wildavsky Jul 03 '20 at 06:25
Ah one thing I should specify is that when you run `python3 -m unittest discover -s project/test` make sure you are in the directory containing `project`. I set up the skeleton for that and it seems to work – kingkupps Jul 03 '20 at 06:30

score 0 · Answer 2 · answered Jul 07 '20 at 02:15

This is what I ended up with, thanks to the suggestions here and one from Alex Martelli. The secret sauce is the use of Python's -m flag.

project
    src
        __init.py__
        file1.py
            class Class1:
                pass
        file2.py
            from src.file1 import Class1
            class Class2:
                pass
            if __name__ == "__main__":
                # Instantiate Class2 here…
                pass
    test
        __init.py__
        test1.py
            from src.file1 import Class1
            class TestClass1:
                pass
        test2.py
            from src.file2 import Class2
            class TestClass2:
                pass

The app is invoked as cd project && python3 -m src.file2

The tests are run as cd project && python3 -m unittest

How to import without using sys.path?

2 Answers2

Using __init__.py

Using setuptools

Using `init.py`

Using `setuptools`