How to import namespaced packages for which sys.path needs to be adjusted?

Question

In a python application I'm working on I would like to dynamically load packages (plugins) based on information provided at runtime (e.g. from a config file).

So, I change sys.path to add paths to the plugins I want to load, this generally works fine, but not if the plugin is a namespaced package inside a namespace which has already been initialized (I assume that is the problem anyway).

Example:

# lib1 contains plugins/__init__.py and plugins/foo/__init__.py
# lib2 contains plugins/__init__.py and plugins/bar/__init__.py
# plugins is a namespace package

import sys

sys.path.append ('lib1')
import plugins.foo

sys.path.append ('lib2')
import plugins.bar

The above code fails with an ImportError, presumably because the "import plugins.foo" line initialized the plugins namespace/package and no further attempt is made to search sys.path for other packages in the namespace.

If I change the code to this:

sys.path.append ('lib1')
sys.path.append ('lib2')
import plugins.foo
import plugins.bar

Both imports work, but I'd like to add to sys.path after the plugins.foo import.

So, my questions are:

Is my assumption correct that the second import fails because of the namespace package?
Is there a work-around?

You really don't need to mess with sys.path at all. Just use packages. For dynamic imports you can call the `__import__` function. Or use the `imp` module. — Keith, Mar 18 '12 at 13:45
If I am following correctly, why not just not make ``plugins`` a package and add ``plugins/lib1`` and ``plugins/lib2`` to the path, then ``import foo`` and ``import bar``? — Gareth Latty, Mar 18 '12 at 13:46
That said, I think your real question is how to make a plugin system in Python - see this: http://stackoverflow.com/questions/7417997/plugin-manager-in-python/7418304#7418304 — Gareth Latty, Mar 18 '12 at 13:47
Also related https://stackoverflow.com/questions/2918898/prevent-python-from-caching-the-imported-modules — dirkbaechle, Mar 19 '18 at 12:57

alexis · Accepted Answer · 2012-03-18T14:03:11.587

I think your diagnosis is right. When you import plugins.foo, python also loads plugins and caches it in sys.modules. I haven't replicated your set-up, but I'd try del sys.modules['plugins'] before you import bar. If that doesn't work, try reloading plugins:

sys.path.append ('lib2')
import plugins
reload(plugins)
import plugins.bar

I must admit that reloading is, by general agreement, not a good idea in running programs. The "right" solution would be to figure out a workflow that lets you set the full sys.path before you start importing from the package. Or perhaps not to spread one package over different places.

I wasn't familiar with reload, that solves my problem. Thanks! — warp, Mar 18 '12 at 14:05

score 1 · Answer 2 · answered Mar 19 '18 at 15:04

This is not an answer, but an extended comment.

The import situation as given in the original question above doesn't only arise when trying to import two different plugins. In our case we're trying to provide a local fallback for the case that a required Python package isn't installed in the current system.

The overall project structure for a MVE could look like this:

.
|-- src
|   `-- balla
|       |-- __init__.py
|       `-- hurga
|           `-- __init__.py
|-- src2
|   `-- balla
|       |-- __init__.py
|       `-- hurga2
|           `-- __init__.py
`-- test.py

, where src/balla/__init__.py and src2/balla/__init__.py have the same source code:

#!/usr/bin/env python
try:
    import pkg_resources
    pkg_resources.declare_namespace(__name__)
except ImportError:
    from pkgutil import extend_path
    __path__ = extend_path(__path__, __name__)

. The two src/balla/hurga/__init__.py and src2/balla/hurga2/__init__.py are empty for now. In our main script test.py:

import sys, os

sys.path.append(os.path.join(os.path.dirname(sys.argv[0]), 'src'))

try:
    import balla.hurga2
except:
    sys.path.insert(0, os.path.join(os.path.dirname(sys.argv[0]), 'src2'))
    import balla.hurga2

we first try to import the package "balla.hurga2" from the system directly, hoping that somebody installed it properly (e.g. via pip). The first sys.path.append in connection with the additional src folder, try to simulate the fact that other packages with the common namespace "balla" are installed (e.g. balla.extended). But balla.hurga2 isn't present, so the declaration of the namespace "balla" is successful, but the actual import isn't. In the except path we now try to remedy this situation by inserting the local path to our fallback package "src2". On the new import balla.hurga2 the old "balla" entry in the sys.modules dict is cached and remains unchanged. This results in the classical error message:

Traceback (most recent call last):
  File "test.py", line 9, in <module>
    import balla.hurga2
ImportError: No module named hurga2

By removing the cached entry for the "balla" package from sys.modules the problem can be solved:

import sys, os

sys.path.append(os.path.join(os.path.dirname(sys.argv[0]), 'src'))

try:
    import balla.hurga2
except:
    if 'balla' in sys.modules:
        del sys.modules['balla']
    sys.path.insert(0, os.path.join(os.path.dirname(sys.argv[0]), 'src2'))
    import balla.hurga2

The additional if-clause is required for systems where no balla package at all is available.

How to import namespaced packages for which sys.path needs to be adjusted?

2 Answers2