How do I dynamically import all *.py files from a given directory AND all sub-directories?

Question

Suppose I have a bunch of Python files underneath one common directory, but within that, an arbitrary number of sub-directories may exist:

/first/foo.py
/first/bar.py
/first/fizz/buzz.py
/first/numbers/one.py
/first/numbers/two.py

I have some arbitrary files where I want to import all of these. Manually, I could do:

import first.foo
import first.bar
import first.fizz.buzz
import first.numbers.one
import first.numbers.two

But instead I'd like to be able to do something like:

import_everything_under('first')

I know that a similar question has already popped up: Recursively import all .py files from all folders

But the given answers and alleged duplicates do NOT answer this question.

---

This question was flagged as a possible duplicate of: How to load all modules in a folder?

Again, that does NOT answer this question. The answer given to that question is not recursive - it would ONLY import items from the immediate directory and does NOT include scripts in sub-directories, which I need for my use-case.

Answer 1

Your problem is broken up into two steps (from my point of view):

1. Walk through your directories and sub-directories and find the names of all the .py files you want to import

2. Import them!

To solve (1), we can use the os.walk method to find all the .py files. This function would look like this:

import os
def get_py_files(src):
    cwd = os.getcwd() # Current Working directory
    py_files = [] 
    for root, dirs, files in os.walk(src):
        for file in files:
            if file.endswith(".py"):
                py_files.append(os.path.join(cwd, root, file))
    return py_files

Now that you have a list of your .py files, we need a function that can dynamically import them.

import importlib
def dynamic_import(module_name, py_path):
    module_spec = importlib.util.spec_from_file_location(module_name, py_path)
    module = importlib.util.module_from_spec(module_spec)
    module_spec.loader.exec_module(module)
    return module

And now we just need to put it all together, writing a function that calls your get_py_files function and then loops over the results, loading the modules with dynamic_import.

I am assuming you want the module names that you use in your python script to be the same as the file name, however you can change that by modifying the module_name variable in the function below.

def dynamic_import_from_src(src, star_import = False):
    my_py_files = get_py_files(src)
    for py_file in my_py_files:
        module_name = os.path.split(py_file)[-1].strip(".py")
        imported_module = dynamic_import(module_name, py_file)
        if star_import:
            for obj in dir(imported_module):
                globals()[obj] = imported_module.__dict__[obj]
        else:
            globals()[module_name] = imported_module
    return

Notice we have to call globals() to add the module to the global namespace. Without doing this, the module will be imported but you will have no way of accessing anything inside it. You can pass star_import = True to dynamic_import_from_src if you want it to be a star import instead. (like from first.foo import *. Note that this may overwrite variables in your namespace but that's one of the cons of using the * import anyways.

Throwing it all in one block so it's easier to see it all at once:

import os
import importlib


def get_py_files(src):
    cwd = os.getcwd() # Current Working directory
    py_files = [] 
    for root, dirs, files in os.walk(src):
        for file in files:
            if file.endswith(".py"):
                py_files.append(os.path.join(cwd, root, file))
    return py_files


def dynamic_import(module_name, py_path):
    module_spec = importlib.util.spec_from_file_location(module_name, py_path)
    module = importlib.util.module_from_spec(module_spec)
    module_spec.loader.exec_module(module)
    return module


def dynamic_import_from_src(src, star_import = False):
    my_py_files = get_py_files(src)
    for py_file in my_py_files:
        module_name = os.path.split(py_file)[-1].strip(".py")
        imported_module = dynamic_import(module_name, py_file)
        if star_import:
            for obj in dir(imported_module):
                globals()[obj] = imported_module.__dict__[obj]
        else:
            globals()[module_name] = imported_module
    return

if __name__ == "__main__":
    dynamic_import_from_src("first", star_import = False)

At this point, you can access any of the modules you imported the exact same way as if you did import first.whatever. For example, if first/numbers/one.py contained x=1, then you would be able to access that by saying one.x.

Answer 2

The solution by @SyntaxVoid should work but in parallel I've been tinkering at this problem myself, and I think I've found another way of doing it. Either my own solution or @SyntaxVoid's solution should work, I believe.

from importlib import util
from os import path
from glob import glob


def import_submodules(start_path, include_start_directory=True):
    start_path = path.abspath(start_path)
    pattern = '**/*.py' if include_start_directory else '*/**/*.py'
    py_files = [f for f in glob(path.join(start_path, pattern), recursive=True) if not f.endswith('__.py')]

    for py_file in py_files:
        spec = util.spec_from_file_location('', py_file)
        module = util.module_from_spec(spec)
        spec.loader.exec_module(module)

Answer 3

I'm sorry. I don't know that this way is alright. But, try that.

from first import *