Plugin architecture - Plugin Manager vs inspecting from plugins import *

Question

I'm currently writing an application which allows the user to extend it via a 'plugin' type architecture. They can write additional python classes based on a BaseClass object I provide, and these are loaded against various application signals. The exact number and names of the classes loaded as plugins is unknown before the application is started, but are only loaded once at startup.

During my research into the best way to tackle this I've come up with two common solutions.

Option 1 - Roll your own using imp, pkgutil, etc.
See for instance, this answer or this one.

Option 2 - Use a plugin manager library
Randomly picking a couple

• straight.plugin
• yapsy
• this approach

My question is - on the proviso that the application must be restarted in order to load new plugins - is there any benefit of the above methods over something inspired from this SO answer and this one such as:

import inspect
import sys
import my_plugins

def predicate(c):
    # filter to classes
    return inspect.isclass(c)

def load_plugins():
    for name, obj in inspect.getmembers(sys.modules['my_plugins'], predicate):
        obj.register_signals()

Are there any disadvantages to this approach compared to the ones above? (other than all the plugins must be in the same file) Thanks!

EDIT
Comments request further information... the only additional thing I can think to add is that the plugins use the blinker library to provide signals that they subscribe to. Each plugin may subscribe to different signals of different types and hence must have its own specific "register" method.

Answer 1

Since Python 3.6 a new class method __init_subclass__ is added, that is called on a base class, whenever a new subclass is created.

This method can further simplify the solution offered by will-hart above, by removing the metaclass.

The __init_subclass__ method was introduced with PEP 487: Simpler customization of class creation. The PEP comes with a minimal example for a plugin architecture:

It is now possible to customize subclass creation without using a metaclass. The new __init_subclass__ classmethod will be called on the base class whenever a new subclass is created:

class PluginBase:
    subclasses = []

    def __init_subclass__(cls, **kwargs):
        super().__init_subclass__(**kwargs)
        cls.subclasses.append(cls)

class Plugin1(PluginBase):
    pass

class Plugin2(PluginBase):
    pass

The PEP example above stores references to the classes in the Plugin.plugins field.

If you want to store instances of the plugin classes, you can use a structure like this:

class Plugin:
    """Base class for all plugins. Singleton instances of subclasses are created automatically and stored in Plugin.plugins class field."""
    plugins = []

    def __init_subclass__(cls, **kwargs):
        super().__init_subclass__(**kwargs)
        cls.plugins.append(cls())

class MyPlugin1(Plugin):
    def __init__(self):
        print("MyPlugin1 instance created")

    def do_work(self):
        print("Do something")

class MyPlugin2(Plugin):
    def __init__(self):
        print("MyPlugin2 instance created")

    def do_work(self):
        print("Do something else")

for plugin in Plugin.plugins:
    plugin.do_work()

which outputs:

MyPlugin1 instance created
MyPlugin2 instance created
Do something
Do something else

Answer 2

The metaclass approach is useful for this issue in Python < 3.6 (see @quasoft's answer for Python 3.6+). It is very simple and acts automatically on any imported module. In addition, complex logic can be applied to plugin registration with very little effort. It requires:

The metaclass approach works like the following:

1) A custom PluginMount metaclass is defined which maintains a list of all plugins

2) A Plugin class is defined which sets PluginMount as its metaclass

3) When an object deriving from Plugin - for instance MyPlugin is imported, it triggers the __init__ method on the metaclass. This registers the plugin and performs any application specific logic and event subscription.

Alternatively if you put the PluginMount.__init__ logic in PluginMount.__new__ it is called whenver a new instance of a Plugin derived class is created.

class PluginMount(type):
    """
    A plugin mount point derived from:
        http://martyalchin.com/2008/jan/10/simple-plugin-framework/
    Acts as a metaclass which creates anything inheriting from Plugin
    """

    def __init__(cls, name, bases, attrs):
        """Called when a Plugin derived class is imported"""

        if not hasattr(cls, 'plugins'):
            # Called when the metaclass is first instantiated
            cls.plugins = []
        else:
            # Called when a plugin class is imported
            cls.register_plugin(cls)

    def register_plugin(cls, plugin):
        """Add the plugin to the plugin list and perform any registration logic"""

        # create a plugin instance and store it
        # optionally you could just store the plugin class and lazily instantiate
        instance = plugin()

        # save the plugin reference
        cls.plugins.append(instance)

        # apply plugin logic - in this case connect the plugin to blinker signals
        # this must be defined in the derived class
        instance.register_signals()

Then a base plugin class which looks like:

class Plugin(object):
    """A plugin which must provide a register_signals() method"""
    __metaclass__ = PluginMount

Finally, an actual plugin class would look like the following:

class MyPlugin(Plugin):
    def register_signals(self):
        print "Class created and registering signals"

    def other_plugin_stuff(self):
        print "I can do other plugin stuff"

Plugins can be accessed from any python module that has imported Plugin:

for plugin in Plugin.plugins:
    plugin.other_plugin_stuff()

See the full working example

Answer 3

The approach from will-hart was the most useful one to me! For i needed more control I wrapped the Plugin Base class in a function like:

def get_plugin_base(name='Plugin',
                       cls=object,
                       metaclass=PluginMount):

    def iter_func(self):
        for mod in self._models:
            yield mod

    bases = not isinstance(cls, tuple) and (cls,) or cls

    class_dict = dict(
        _models=None,
        session=None
    )

    class_dict['__iter__'] = iter_func

    return metaclass(name, bases, class_dict)

and then:

from plugin import get_plugin_base
Plugin = get_plugin_base()

This allows to add additional baseclasses or switching to another metaclass.