Dynamically loading objects from list of python files

Question

I'm wondering if I have some python scripts in a folder that will have common functionality like method names but different functionality can by dynamically imported. These python scripts will be in a single folder but I don't know how many their will be. I'd like to have a script look in the folder, get a list of the scripts and create objects for each of them.

Answer 1

What you're describing is what's usually called a plugin system.

For Python 3.4 and later, there's a standard recipe for importing a source file.

The only things left to do are (a) enumerating all the source files in a specified directory, and (b) calling some factory/register/whatever function on each module after you've loaded it.

So:

def load_module(path):
    name = os.path.split(path)[-1]
    spec = importlib.util.spec_from_file_location(name, path)
    module = importlib.util.module_from_spec(spec)
    spec.loader.exec_module(module)
    return module

plugins = []
for fname in os.listdir(PLUGIN_DIR):
    if not fname.startswith('.') and fname.endswith('.py'):
        try:
            module = load_module(os.path.join(PLUGIN_DIR, fname)
            plugins.append(module.plugin_factory())

The plugin_factory is just a function or class that "creates an object" for each module. You can call it whatever you want. Often, either every plugin provides a class that's a subclass of (or duck-types as) some type that you defined, or provides a function that returns an instance of some class that you defined. But in really simple cases, you can just use the module itself as the plugin object. In that case, of course, the last line just becomes plugins.append(module).

And obviously, if you want to access the plugins by name rather than just iterating over them (and they don't have some name attribute), you'll want to store them in a dict rather than a list.

And of course you can add them to sys.modules, as in the recipe in the docs, but usually you don't need that.

Answer 2

Suppose we have three files in current dir.

a.py:

def f():
    pass

b.py:

def g():
    pass

c.py:

def h():
    pass

And the following main.py script which scans current directory for python files, and imports them as needed:

import os

modules = {}
for f in os.listdir('.'):
    if os.path.isfile(f) and f.endswith('.py') and f != 'main.py':
        modname = f[:-3] # remove '.py' extension
        modules[modname] = __import__(modname)

# now the various functions are available via:
modules['a'].f()
modules['b'].g()
modules['c'].h()

In Python >=3.4 you can use the importlib module, and avoid __import__, whose use is discouraged in the python docs:

import os
import importlib

modules = {}
for f in os.listdir('.'):
    if os.path.isfile(f) and f.endswith('.py') and f != 'main.py':
        modname = f[:-3] # remove '.py' extension
        spec = importlib.util.spec_from_file_location(modname, f)
        modules[modname] = importlib.util.module_from_spec(spec)
        spec.loader.exec_module(modules[modname])

modules['a'].f()
modules['b'].g()
modules['c'].h()