Dynamic importing of modules followed by instantiation of objects with a certain baseclass from said modules

Question

I'm writing an application. No fancy GUI:s or anything, just a plain old console application. This application, lets call it App, needs to be able to load plugins on startup. So, naturally, i created a class for the plugins to inherit from:

class PluginBase(object):
    def on_load(self):
        pass
    def on_unload(self):
        pass
    def do_work(self, data):
        pass

The idea being that on startup, App would walk through the current dir, including subdirs, searching for modules containing classes that themselves are subclasses of PluginBase.

More code:

class PluginLoader(object):
    def __init__(self, path, cls):
        """ path=path to search (unused atm), cls=baseclass """
        self.path=path
    def search(self):
        for root, dirs, files in os.walk('.'):
            candidates = [fname for fname in files if fname.endswith('.py') \
                                    and not fname.startswith('__')]
        ## this only works if the modules happen to be in the current working dir
        ## that is not important now, i'll fix that later
        if candidates:
            basename = os.path.split(os.getcwd())[1]
            for c in candidates:
                modname = os.path.splitext(c)[0]
                modname = '{0}.{1}'.format(basename, modname)
                __import__(mod)
                module = sys.modules[mod]

After that last line in search I'd like to somehow a) find all classes in the newly loaded module, b) check if one or more of those classes are subclasses of PluginBase and c) (if b) instantiate that/those classes and add to App's list of loaded modules.

I've tried various combinations of issubclass and others, followed by a period of intense dir:ing and about an hour of panicked googling. I did find a similar approach to mine here and I tried just copy-pasting that but got an error saying that Python doesn't support imports by filename, at which point I kind of lost my concentration and as a result of that, this post was written.

I'm at my wits end here, all help appreciated.

Answer 1

You might do something like this:

for c in candidates:
    modname = os.path.splitext(c)[0]
    try:
        module=__import__(modname)   #<-- You can get the module this way
    except (ImportError,NotImplementedError):
        continue
    for cls in dir(module):          #<-- Loop over all objects in the module's namespace
        cls=getattr(module,cls)
        if (inspect.isclass(cls)                # Make sure it is a class 
            and inspect.getmodule(cls)==module  # Make sure it was defined in module, not just imported
            and issubclass(cls,base)):          # Make sure it is a subclass of base
            # print('found in {f}: {c}'.format(f=module.__name__,c=cls))
            classList.append(cls)

To test the above, I had to modify your code a bit; below is the full script.

import sys
import inspect
import os

class PluginBase(object): pass

def search(base):
    for root, dirs, files in os.walk('.'):
        candidates = [fname for fname in files if fname.endswith('.py') 
                      and not fname.startswith('__')]
        classList=[]
        if candidates:
            for c in candidates:
                modname = os.path.splitext(c)[0]
                try:
                    module=__import__(modname)
                except (ImportError,NotImplementedError):
                    continue
                for cls in dir(module):
                    cls=getattr(module,cls)
                    if (inspect.isclass(cls)
                        and inspect.getmodule(cls)==module
                        and issubclass(cls,base)):
                        # print('found in {f}: {c}'.format(f=module.__name__,c=cls))
                        classList.append(cls)
        print(classList)

search(PluginBase)

Answer 2

You would make this a lot easier if you forced some constraints on the plugin writer, for example that all plugins must be packages that contain a load_plugin( app, config) function that returns a Plugin instance. Then all you have to do is try to import these packages and run the function.

Answer 3

Here is a meta-classier way to register the plugins:

Define PluginBase to be of type PluginType. PluginType automatically registers any instance (class) in the plugins set.

plugin.py:

plugins=set()
class PluginType(type):
    def __init__(cls, name, bases, attrs):
        super(PluginType, cls).__init__(name, bases, attrs)
        # print(cls, name,cls.__module__)
        plugins.add(cls)

class PluginBase(object):
    __metaclass__=PluginType
    pass

This is the part that the user writes. Notice that there is nothing special here.

pluginDir/myplugin.py:

import plugin
class Foo(plugin.PluginBase):
    pass

Here is what the search function might look like:

test.py:

import plugin
import os
import imp

def search(plugindir):
    for root, dirs, files in os.walk(plugindir):
        for fname in files:
            modname = os.path.splitext(fname)[0]
            try:
                module=imp.load_source(modname,os.path.join(root,fname))
            except Exception: continue

search('pluginDir')
print(plugin.plugins)

Running test.py yields

set([<class 'myplugin.Foo'>])

Answer 4

Could you use execfile() instead of import with a specified namespace dict, then iterate over that namespace with issubclass, etc?