Skip directory name in import path by importing subpackage in __init__.py

Question

I'm baffled by the importing dynamics in __init__.py. Say I have this structure:

package
├── __init__.py
└── subpackage
    ├── __init__.py
    └── dostuff.py

I would like to import things in dostuff.py. I could do it like this: from package.subpackage.dostuff import thefunction, but I would like to remove the subpackage level in the import statement, so it would look like this:

from package.dostuff import thefunction

I tried putting this in package/__init__.py:

from .subpackage import dostuff

And what I don't understand is this:

# doing this works:
from package import dostuff
dostuff.thefunction()

# but this doesn't work:
from package.dostuff import thefunction
# ModuleNotFoundError: No module named 'package.dostuff'

Why is that, and how can I make from package.dostuff import thefunction work?

Answer 1

The only way I see to make what you intend would be to actually create a package/dostuff.py module and import all you need in it as from .subpackage.dostuff import thefunction.

The point is that when you use from .subpackage import dostuff in package/__init__.py, you do not rename the original module.

To be more explicit, here is an example of use with both your import and a package/dostuff.py file:

# We import the dostuff link from package
>>> from package import dostuff
>>> dostuff
<module 'package.subpackage.dostuff' from '/tmp/test/package/subpackage/dostuff.py'>

# We use our custom package.dostuff
>>> from package.dostuff import thefunction
>>> package.dostuff
<module 'package.dostuff' from '/tmp/test/package/dostuff.py'>
>>> from package import dostuff
>>> dostuff
<module 'package.dostuff' from '/tmp/test/package/dostuff.py'>

# The loaded function is the same
>>> dostuff.thefunction
<function thefunction at 0x7f95403d2730>
>>> package.dostuff.thefunction
<function thefunction at 0x7f95403d2730>

A clearer way of putting this is:

from X import Y only works when X is an actual module path. Y on the contrary can be any item imported in this module.

This also applies to packages with anything being declared in their __init__.py. Here you declare the module package.subpackage.dostuff in package, hence you can import it and use it.

But if you try to use the module for a direct import, it has to exist on the filesystem

Resources:

• Python documentation about module management in the import system:
  • https://docs.python.org/3/reference/import.html#submodules.
• Python import system search behavior:
  • https://docs.python.org/3/reference/import.html#searching
  • https://docs.python.org/3/glossary.html#term-qualified-name
  • https://docs.python.org/2.0/ref/import.html

I hope that makes it clearer

Answer 2

You can in fact fake this quite easily by fiddling with Python's sys.modules dict. The question is whether you do really need this or whether it might be good to spend a second thought on your package structure.

Personally, I would consider this bad style, because it applies magic to the module and package names and people who might use and extend your package will have a hard time figuring out what's going on there.

Following your structure above, add the following code to your package/__init__.py:

import sys

from .subpackage import dostuff

# This will be package.dostuff; just avoiding to hard-code it.
_pkg_name = f"{__name__}.{dostuff.__name__.rsplit('.', 1)[1]}"

if _pkg_name not in sys.modules.keys():
    dostuff.__name__ = _pkg_name  # Will have no effect; see below
    sys.modules[_pkg_name] = dostuff

This imports the dostuff module from your subpackage to the scope of package, changes its module path and adds it to the imported modules. Essentially, this just copies the binding of your module to another import path where member memory addresses remain the same. You just duplicate the references:

import package
print(package.dostuff)
print(package.subpackage.dostuff)
print(package.dostuff.something_to_do)
print(package.subpackage.dostuff.something_to_do)

... yields

<module 'package.subpackage.dostuff' from '/path/package/subpackage/dostuff.py'>
<module 'package.subpackage.dostuff' from '/path/package/subpackage/dostuff.py'>
<function something_to_do at 0x1029b8ae8>
<function something_to_do at 0x1029b8ae8>

Note that

1. The module name package.subpackage.dostuff has not changed even though being updated in package/__init__.py
2. The function reference is the same: 0x1029b8ae8

Now, you can also go

from package.dostuff import something_to_do
something_to_do()

However, be cautious. Changing the imported modules during import of a module might have unintended side-effects (also the order of updating sys.modules and importing other subpackages or submodules from within package might be relevant). Usually, you buy extra work and extra complexity by applying such kind of "improvement". Better yet set up a proper package structure and stick to it.