Importing dynamically all modules from a folder

Question

I know there is this question, but not only they're not working, it's not exactly what I want. I'm developing a racing game and want to load all tracks from a folder dynamically (They're stored as .py instead of .json). I don't want to know the names of the tracks, since users can mod/add them at will. I just want to import their data. So, for example:

>tracks 
>>track0.py
>>track1.py
>>track2.py
>>track3.py
>>track4.py

Inside each track, I have data like this:

track_ground_data = [
    [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
    [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
    [1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 1],
    [1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 1],
    [1, 1, 2, 2, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 1, 1],
    [1, 1, 2, 2, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 1, 1],
    [1, 1, 2, 2, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 1, 1],
    [1, 1, 2, 2, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 1, 1],
    [1, 1, 2, 2, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 1, 1],
    [1, 1, 2, 2, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 1, 1],
    [1, 1, 2, 2, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 1, 1],
    [1, 1, 2, 2, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 1, 1],
    [1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 1],
    [1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 1],
    [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
    [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
]

I need to import each track module like this:

loaded_tracks = [t for t in tracks] # Where tracks is the folder.

And then access a given track_ground_data like this:

loaded_tracks[0].track_ground_data

If I knew Python was going to be so harsh with its imports, I'd have used json instead .py.

Yes. I've placed that code inside a `__init__.py` file and when I try to import a track (`import tracks`): `tracks.track0`, I get an `AttributeError`. — Ericson Willians, Oct 27 '17 at 17:43
You're right, you probably should have used a file format designed for storing data instead of executable .py files... — martineau, Oct 27 '17 at 17:50
You could do this by adapting [my answer](https://stackoverflow.com/a/14428820/355230) to a related question about packages. — martineau, Oct 27 '17 at 17:52
Using python files for tracks is useless if they should only contain *static* data. However they do allow to put arbitrary code, which may be a feature... for example some track could define some functions that provide specific logic for that particular track, this is way harder to do with a simple json. If you don't need the flexibility just use json files (btw: you could still import them as python modules by using a custom module loader...) — Bakuriu, Oct 27 '17 at 17:56

score 6 · Accepted Answer · edited Aug 31 '18 at 21:03

Python does not automatically import submodules contained in a package. Hence import tracks only loads tracks/__init__.py.

However you can put code inside the __init__.py file that imports all the modules it finds in that directory.

For example putting something like this in the __init__.py:

import os
import importlib

__globals = globals()

for file in os.listdir(os.path.dirname(__file__)):
    mod_name = file[:-3]   # strip .py at the end
    __globals[mod_name] = importlib.import_module('.' + mod_name, package=__name__)

Should make your submodules available as tracks.trackX when importing only tracks.

Or you could use exec:

import os
import importlib

for file in os.listdir(os.path.dirname(__file__)):
    mod_name = file[:-3]   # strip .py at the end
    exec('import .' + mod_name)

A cleaner approach would be to use import hooks or implement your own custom module importer. There are multiple ways to do this using importlib see also sys.path_hooks

You'd also want to filter out `__init__.py`, so only do the last statement if `mod_name` doesn't match the pattern `^__`. Something like `if not re.match(r'^__', file):`. — superhawk610, Aug 31 '18 at 19:21

score 2 · Answer 2 · answered Oct 27 '17 at 19:36

Just for the sake of the good wellfare of future pythoners, I'm posting how I've solved. A friend helped me through it. Coudln't make Bakuriu's solution work, because the modules came empty. Inside __init__.py I've put:

import os

dir = os.path.dirname(os.path.abspath(__file__))
modules = [os.path.splitext(_file)[0] for _file in os.listdir(dir) if not _file.startswith('__')]

tracks = []
for mod in modules:
    exec('from tracks import {}; tracks.append({})'.format(mod, mod))

And then, on the main file, I've loaded it as:

dir = os.path.dirname(os.path.abspath(__file__))
sys.path.append(dir)

from tracks import tracks

And then:

loaded_tracks = [t for t in tracks]

That actually solved it quite well. I was almost switching to JSON / giving up.

score 1 · Answer 3 · answered Oct 27 '17 at 17:51

The problem of dynamically importing modules is faced usually when frameworks have a plug-in or add on system for the community to contribute. Each plug-in or add-on is a module containing classes and functions compliant with the framework's architecture and api.

With that in mind, the solution for "joining the dots" between the framework code and arbitrarily many add-ons is through the importlib present in the python standard library. You seem to face the same structural problem.

Here is a stackoverflow question that was answered with importlib. And the documentation.

Importing dynamically all modules from a folder

3 Answers3