Calling a particular function for every module in a python directory

Question

None of the answers on stack overflow have helped me with this problem: Load module from string in python

My file structure is as follows: -run.py -/boards -Brd1.py -Brd2.py -Brd3.py

What I am trying to accomplish is to call the "run" function for every module in the boards directory. Each module has this "run" function. Here is my code in run.py:

import os
import sys,imp
import inspect

for item in os.listdir("boards"):
    if item.startswith("Brd") and item.endswith(".py"):
        curr_module = imp.new_module(item)
        curr_module.run()

This is the error I get:

AttributeError: 'module' object has no attribute 'run'

However, when I explicitly do something like this:

from boards import Brd1

Brd1.run()

It works perfectly. Interestingly enough, the code:

import os
import sys,imp
import inspect

for item in os.listdir("boards"):
    if item.startswith("Brd") and item.endswith(".py"):
        curr_module = imp.new_module(item)
        print dir(curr_module)

just prints: ['__ doc __ ', ' __ name __ ', '__ package __ ']

Is there anyway to do what I am trying to accomplish? Am I going about this the right away?

"None of the answers on stack overflow have helped me with this problem" doesn't really help us. What questions and answers did you look at? How did they not help? — abarnert, Jul 19 '18 at 22:49

score 1 · Answer 1 · edited Jun 20 '20 at 09:12

You're not doing anything to load the modules, you're just creating and running empty modules.

curr_module = imp.new_module(item)
curr_module.run()

From the docs says:

imp.new_module(name)

Return a new empty module object called name. This object is not inserted in sys.modules.

So, this is effectively the same as if you'd written this:

curr_module = types.ModuleType(item)
curr_module.run()

The reason you get ['__ doc __ ', ' __ name __ ', '__ package __ '] (although I'm pretty sure you actually get ['__doc__ ', '__name__', '__package__'] without all those extra spaces) is that those attributes exist even on an empty module.

The example code that you copied this from is trying to do something completely different. The whole point of that question is how to create a module out of a string of source code, instead of the normal way. Which you do (in 2.x) by creating a new empty module, then doing an exec against that module's globals. The exec is the reason the module ends up not being empty, and you're not doing anything similar.

You're also giving the module an invalid name, like Brd1.py instead of Brd1.

If you want to actually load the modules, look at the Examples:

fp, pathname, description = imp.find_module(name)
curr_module = imp.load_module(name, fp, pathname, description)
# …

Of course you don't want to load it by normal search, but by pathname. You could pass a path argument to find_module:

name = os.path.splitext(item)[0]
fp, pathname, description = imp.find_module(name, 'boards')

… or just load it directly:

name = os.path.splitext(item)[0]
pathname = os.path.join("boards", item)
desc = [s for s in imp.get_suffixes if s[0] == '.py']
with open(pathname) as f:
    curr_module = imp.load_module(name, f, pathname, desc)

You might also want to put the module in sys.modules. Or you might not. (The example doesn't, because it's simulating the effects of the __import__ function.) If so:

sys.modules[name] = curr_module

You also might want to do the sys.modules check, as in the example. If you do the check, trying to import the same file twice will use the cached version the second time, as if you'd done a normal import; if you don't, it will be similar to calling reload:

try:
    curr_module = sys.modules[name]
except KeyError:
    # the logic above to find and load curr_module

This doesn't work. I get an ImportError: No module named Brd1 — kaaazemian, Jul 20 '18 at 18:50
@kaaazemian "I get an ImportError" isn't something that can be debugged. Which of the three options above did you try? What does your code look like? Where did the exception come from? — abarnert, Jul 20 '18 at 18:53

kay · Accepted Answer · 2018-07-20T19:00:53.577

This is probably not working for you because as abarnert pointed out, you're only loading an empty module. Also, looking at your file structure, you need to insert the path name of the boards folder:

sys.path.insert(0, './boards')

What you're attempting can be solved by simply using the python documentation for the import internals: https://docs.python.org/2/library/imp.html#imp.get_suffixes

I just tested it and this code does exactly what you're asking for:

for item in os.listdir("boards"):
    if item.startswith("Brd") and item.endswith(".py"):
        name = os.path.splitext(item)[0]
        fp, pathname, description = imp.find_module(name)
        with open(pathname) as f:
            curr_module = imp.load_module(name, fp, pathname, description)
            curr_module.run()

Calling a particular function for every module in a python directory

2 Answers2