Proper package organization for pypi/pip import and for locally running scripts

Question

I have a package I've been developing that I'm having trouble properly organizing in order for my scripts to work both after PyPI/pip install and also locally running it.

I primarily have two files, e.g. my-package.py and utils.py I'm looking to be import-able, one of which references the other. My desired file organization would be:

    LICENSE
    pyproject.toml
    README.md
    my-package/
        __init__.py
        my-package.py
        utils.py
    tests/

I would like to be able to pip install my-package and then import this package in a new script like so: from my-package import my-function (or if I have to, from my-package.my-package import my-function) and also from my-package.utils import util-function

But I also import a util-function from utils.py within my-package.py, and so far I've only been able to get it working for pip OR local, but not both.

E.g., This works after pip installing from my-package.my-package import my-function and also from my-package.utils import util-function. But when I locally try to run python my-package.py, I get the error: ModuleNotFoundError: No module named 'my-package.utils'; 'my-package' is not a package.

I've tried some variations with from .my-package or from .utils but these also have relative import errors.

I might be asking for too much here, but hoping theres some relatively painless guidance I could follow re: organization + importing that would make this work for both post-pip importing and for locally running the scripts?

Thanks for all the help!

To dispel another common point of confuse, the name you put after `pip install` and the name you put after `import` are _unrelated_. The former is the name of the _distribution package_ (nowadays just called "distribution" to avoid confusion) while the latter is the name of the _import package_. — Brian61354270, Jul 20 '23 at 00:59

wim · Accepted Answer · 2023-07-20T01:08:16.430

Do not name the top-level package (i.e. the directory containing the __init__.py file) exactly the same as a submodule name contained within. It's terribly confusing and provides no benefit.

Otherwise, this is a good structure:

LICENSE
pyproject.toml
README.md
my_package/
    __init__.py
    my_submodule.py
    utils.py
tests/

In order for a function defined in my_submodule.py to be importable from the top-level namespace, you need something like this in the __init__.py:

# my_package/__init__.py
from my_package.my_submodule import my_function

This "pulls up" the name my_function from my_package.my_submodule.__dict__ into my_package.__dict__.

But when I locally try to run python my-package.py, I get the error: ModuleNotFoundError: No module named 'my-package.utils'; 'my-package' is not a package.

This is Relative imports for the billionth time. The root cause is actually unrelated to relative imports; the issue exists when using absolute imports too. Your particular error message is different, because in your case the submodule has been imported instead of the top-level package of the same name (that probably means you were inside the package directory when running this command). That's one of the reasons to name the package and the submodule differently, to avoid import confusion.

In all cases the answer is the same: submodules are not scripts. Don't try to run a submodule as a script, no amount of hacking the sys.path will make it work sanely. Running a submodule of a package directly as a script is not well-supported in Python, Guido considers that an anti-pattern. Instead, the standard solution is to declare console_script entry points for functions defined within the package. Wrapper scripts for the entry points will be auto-generated by pip at install time. See my answer here for detailed instructions on how to create the entrypoints.

Proper package organization for pypi/pip import and for locally running scripts

1 Answers1