Python dynamic loads modules/packages conflict

Question

I want to load python(2.7 not 3.x) modules from source code dynamically using imp.load_source and if I dont't append the source code path to sys.path the local import can't work. In this case, How to resolve the conflict module name via local import?

I have been search a while and can't find a solution for this. Any ideas? Thanks!

The file structure what I have like this:

/tmp
|---runner
|   |-runner.py

/tmp/modules
|---bundle1
|   |--bundle
|   | |-__init__.py
|   | |-index.py
|   |-main.py
|
|---bundle2
    |--bundle
    | |-__init__.py
    | |-index.py
    |-main.py

In runner.py

import os
import imp
import sys

sys.path.append("/tmp/modules/bundle1")
module_a = imp.load_source("Bundle1", "/tmp/modules/bundle1/main.py")
sys.path.append("/tmp/modules/bundle2")
module_b = imp.load_source("Bundle2", "/tmp/modules/bundle2/main.py")

module_a.start()
module_b.start()

In bundle1/main.py

from bundle import index
def start():
    index.a_func()

if __name__ == "__main__":
    start()

In bundle2/main.py

from bundle import index
def start():
    index.b_func()

if __name__ == "__main__":
    start()

Because the conflict of from bundle, b_func() can't be found in bundle2/main.py

I'm not sure but maybe you should take a look at the `setuptools` package ? — FunkySayu, Jul 24 '15 at 11:04

score 0 · Answer 1 · edited May 23 '17 at 10:24

What's happening here is that imports are being mixed up. Python is caching the fact that it knows about a module called bundle as well as a module called bundle.index. This post should help you understand imports a little more. This becomes a little less mystifying if you print out sys.modules['bundle'] in between your loads.

print sys.modules.get('bundle', None)
# => yields None

sys.path.append("/tmp/modules/bundle1")
module_a = imp.load_source("Bundle1", "/tmp/modules/bundle1/main.py")
print sys.modules.get('bundle', None)
# => <module 'bundle' from '/tmp/modules/bundle1/bundle/__init__.pyc>

sys.path.append("/tmp/modules/bundle2")
module_b = imp.load_source("Bundle2", "/tmp/modules/bundle2/main.py")
print sys.modules.get('bundle', None)
# => <module 'bundle' from '/tmp/modules/bundle1/bundle/__init__.pyc>

If you have some real need for loading the exact same module name twice in one namespace, you'll need to blow away the entry in sys.modules in between. In addition, since you're just appending to your path, when you load module_b python will find /tmp/modules/bundle1/bundle/ before it will find /tmp/modules/bundle2/bundle So, to completely solve you're problem you'll have to either prepend your information or remove it from the sys.path In general though, this really isn't great practice and certainly if you have any control over the module names, you should change them.

print sys.modules.get('bundle', None)
# => yields None

sys.path.append("/tmp/modules/bundle1")
module_a = imp.load_source("Bundle1", "/tmp/modules/bundle1/main.py")
print sys.modules.get('bundle', None)
# => <module 'bundle' from '/tmp/modules/bundle1/bundle/__init__.pyc>

module_a.start()

# Delete python's knowledge of bundle1
del sys.modules['bundle']
del sys.modules['bundle.index']
sys.path.pop(sys.path.index('/tmp/modules/bundle1')

sys.path.append("/tmp/modules/bundle2")
module_b = imp.load_source("Bundle2", "/tmp/modules/bundle2/main.py")
print sys.modules.get('bundle', None)
# => <module 'bundle' from '/tmp/modules/bundle2/bundle/__init__.pyc>

module_b.start()

Please note that since you deleted python's knowledge of bundle1, that attempting: moudle_a.start() after the del will result in an AttributeError

Hope this helps.

Thanks for your response. Yeah, I know that will mixed up `sys.modules`... Is there any isolation/container tricks? — YuLun, Jul 27 '15 at 02:02
Not that I am aware of, short of what I posted above. You could certainly write your own hacks though. Perhaps consider using a class through the `with` syntax? — loganasherjones, Jul 27 '15 at 16:10

Python dynamic loads modules/packages conflict

1 Answers1