python: dynamically loading one-time plugins?

Question

I'm writing a python application in which I want to make use of dynamic, one-time-runnable plugins.

By this I mean that at various times during the running of this application, it looks for python source files with special names in specific locations. If any such source file is found, I want my application to load it, run a pre-named function within it (if such a function exists), and then forget about that source file.

Later during the running of the application, that file might have changed, and I want my python application to reload it afresh, execute its method, and then forget about it, like before.

The standard import system keeps the module resident after the initial load, and this means that subsequent "import" or "__import__" calls won't reload the same module after its initial import. Therefore, any changes to the python code within this source file are ignored during its second through n-th imports.

In order for such packages to be loaded uniquely each time, I came up with the following procedure. It works, but it seems kind of "hacky" to me. Are there any more elegant or preferred ways of doing this? (note that the following is an over-simplified, illustrative example)

import sys
import imp
# The following module name can be anything, as long as it doesn't
# change throughout the life of the application ...
modname = '__whatever__'
def myimport(path):
    '''Dynamically load python code from "path"'''
    # get rid of previous instance, if it exists
    try:
        del sys.modules[modname]
    except:
        pass
    # load the module
    try:
        return imp.load_source(modname, path)
    except Exception, e:
        print 'exception: {}'.format(e)
        return None

 mymod = myimport('/path/to/plugin.py')
 if mymod is not None:
     # call the plugin function:
     try:
         mymod.func()
     except:
         print 'func() not defined in plugin: {}'.format(path)

Addendum: one problem with this is that func() runs within a separate module context, and it has no access to any functions or variables within the caller's space. I therefore have to do inelegant things like the following if I want func_one(), func_two() and abc to be accessible within the invocation of func():

def func_one():
    # whatever

def func_two():
    # whatever

abc = '123'

# Load the module as shown above, but before invoking mymod.func(),
# the following has to be done ...

mymod.func_one = func_one
mymod.func_two = func_two
mymod.abc      = abc

# This is a PITA, and I'm hoping there's a better way to do all of
# this.

Thank you very much.

Although it is a bit hacky, I don't see anything wrong with your approach. In terms of your code, though, you should never catch all exceptions with `except:`. Instead, catch specific exceptions, such as `except KeyError:` or `except IOError:`. — Rushy Panchal, May 15 '16 at 01:07
Thank you. My code is oversimplified, as I stated. In the real world, my error checking is much more extensive. But also, read my addendum, which I added after you wrote your comment. — HippoMan, May 15 '16 at 01:14
Why not using the `reload(module)` builtin function? https://docs.python.org/2/library/functions.html#reload — gdlmx, May 15 '16 at 01:25
Aha! reload(module) is what I was looking for. However, it still doesn't address the concerns I raised in my addendum. Perhaps I have to live with those limitations, though ... ??? Ideally, I'd like to dynamically load my dynamic python functions into the \_current\_ name space. — HippoMan, May 15 '16 at 01:32

gdlmx · Answer 1 · 2016-05-15T01:53:07.933

2

The approach you use is totally fine. For this question

one problem with this is that func() runs within a separate module context, and it has no access to any functions or variables within the caller's space.

It may be better to use execfile function:

# main.py
def func1():
  print ('func1 called')
exec(open('trackableClass.py','r').read(),globals()) # this is similar to import except everything is done in the current module
#execfile('/path/to/plugin.py',globals())  # python 2 version
func()

Test it:

#/path/to/plugin.py
def func():
  func1()

Result:

python main.py
# func1 called

One potential problem with this approach is namespace pollution because every file is run in the current namespace which increase the chance of name conflict.

edited May 15 '16 at 01:53

answered May 15 '16 at 01:34

gdlmx

6,479
1
21
39

I previously had problems with execfile(), and that's what led me to the method I came up with. But perhaps those were coding problems on my part. I'll do some testing and come back here soon with either an explanation of my execfile() problems, or a declaration that the problem has been solved. – HippoMan May 15 '16 at 01:36
execfile is not available in 3.x; I needed to load the file, compile then exec. Maybe six has a compatibility function. – Neapolitan May 15 '16 at 01:39
... and yes, I now rediscovered the execfile() problem I had. The 2nd through n-th invocations of execfile() do not override the load from the initial execfile call(). In other words, if the file is loaded via execfile() and func() returns 1 initially, and then if I change the file so func() returns 2, a subsequent execfile() call still results in func() returning 1. – HippoMan May 15 '16 at 01:40
That's quite surprising, because I test it on my machine with winPython2.7.10 and don't have such problem – gdlmx May 15 '16 at 01:45
You can find a equivalent method in python 3 to execfile here : http://stackoverflow.com/questions/436198/what-is-an-alternative-to-execfile-in-python-3-0 – gdlmx May 15 '16 at 01:48
No, I'm wrong. I mis-coded my test script. Subsequent calls to execfile() do indeed capture the latest version of the code within the file that's loaded. However, Neapolitian's comment reminds me of why I didn't use execfile() ... it's because I am planning to convert my application to python3. – HippoMan May 15 '16 at 01:48
Now my code works for both python 2 and 3, Neapolitian already show how to separate the namespace so I won't repeat here. – gdlmx May 15 '16 at 01:58

Neapolitan · Accepted Answer · 2016-05-15T01:55:15.000

2

I use the following code to do this sort of thing.

Note that I don't actually import the code as a module, but instead execute the code in a particular context. This lets me define a bunch of api functions automatically available to the plugins without users having to import anything.

def load_plugin(filename, context):
    source = open(filename).read()
    code = compile(source, filename, 'exec')
    exec(code, context)
    return context['func']

context = { 'func_one': func_one, 'func_two': func_two, 'abc': abc }
func = load_plugin(filename, context)
func()

This method works in python 2.6+ and python 3.3+

edited May 15 '16 at 01:55

answered May 15 '16 at 01:37

Neapolitan

2,101
9
21

This is what I need. Thank you, Neapolitan. And it's more "elegant" than my own, hacky solution. – HippoMan May 15 '16 at 01:58
PS: ... and if I want my plugin function to have access to everything in my current namespace, I presume I could just do this: load_plugin(filename, globals()) ... correct? – HippoMan May 15 '16 at 02:05
You have to be a bit careful with `globals()`. If I do `s=5;globals()['s']=6` then `s` will be 6 in the caller's namespace. That is, in cpython at least, one plugin could potentially mess up the namespace of another plugin. If you instead use `context = globals().copy()` then you can pull the symbols you want out of context when the plugin is finished without affecting your namespace or the namespace of other plugins. – Neapolitan May 15 '16 at 02:30
Yes, I see that globals().copy() is necessary. Thank you. – HippoMan May 15 '16 at 02:42

python: dynamically loading one-time plugins?

2 Answers2