Deeply nested namespaced packages

Question

I need to maintain some ETL tool constructed in such a way tasks and pipelines are defined as collection of python packages. Think about plugin architecture with small core and almost thousand of plugins in nested namespaces/packages/subpackages. This is not a hot mess yet, overall quality is quite good, but setup.py and __init__.py-s look very hacky and sometimes cause unexpected problems during imports.

I would like to simplify this a bit. Since Python 3.3 we can put packages in namespaces simply by creating subdirectories without __init__.py. This is exactly what I need, but I would like to avoid deeply nested subdirectories in source code, because a large amount of packages is very small. In extreme they would look like this:

$ tree
.
├── setup.cfg
├── setup.py
└── src
    └── foo
        └── bar
            └── baz
                └── xyz
                    └── uvw
                        └── package
                            ├── actual_code.py
                            └── __init__.py

Is there a way to use implicit namespaces without so deep structure and simply specify namespace somewhere in setup.py (or even better setup.cfg)? In other words, is there a simple way to tell: install package in foo.bar.baz.xyz.uvw namespace?

I would like to have structure like this:

$ tree
.
├── setup.cfg
├── setup.py
└── src
    └── package
        ├── actual_code.py
        └── __init__.py

but installation process should put package into foo/bar/baz/xyz/uvw/package folder, so it could be imported with full path.

Edit: Is this even a good idea?

Have you looked into specifying `__all__`? [example](https://stackoverflow.com/questions/30888683/why-should-i-use-all-in-init-of-python-package) — MyNameIsCaleb, Sep 24 '19 at 17:48
This is not about making long imports short. I need to have that deep structure _after_ installation, but I don't want it in source code repository. — maln0ir, Sep 24 '19 at 18:08

ShadowRanger · Accepted Answer · 2019-09-24T19:37:29.557

4

This is possible using the package_dir argument to distutils.core.setup (or equivalent from setuptools).

Just modify your setup.py to contain something like:

from distutils.core import setup

setup(# ... other setup arguments ...
      package_dir={'foo.bar.baz.xyz.uvw': 'src'},
      packages=['foo.bar.baz.xyz.uvw.package'],
     )

The key part here is that package_dir is saying "the contents of foo.bar.baz.xyz.uvw are what is found in the src directory", while packages=['foo.bar.baz.xyz.uvw.package'] tells it to expect to find and install a package named foo.bar.baz.xyz.uvw.package.

The setup.cfg equivalent would be:

[options]
package_dir=
    foo.bar.baz.xyz.uvw=src
packages = foo.bar.baz.xyz.uvw.package

edited Sep 24 '19 at 19:37

answered Sep 24 '19 at 19:31

ShadowRanger

143,180
12
188
271

1

FYI, I couldn't find a convenient way of using `setuptools.find_namespace_packages` (or the `setup.cfg` approach of `packages=find_namespace:` directive, with `[option.packages.find]` using the `where` key) to avoid needing to manually list the contents of `packages`, so this does end up a little redundant (`foo.bar.baz.xyz.uvw` has to be typed multiple times). If someone knows how to do this, please chime in; `find_namespace_packages` could be used with a list comprehension to programmatically fix up the prefix, but an existing API that does it for you would be preferable. – ShadowRanger Sep 24 '19 at 19:46
Oh, so this is how it works. This is almost perfect solution, thank you! – maln0ir Sep 24 '19 at 20:19
Repetition is not a big problem, because I also wanted to create single template for all packages. It seems to be possible now (and quite simple). – maln0ir Sep 24 '19 at 20:26

Deeply nested namespaced packages

1 Answers1