How do I load all modules under a subdirectory in Python?

Question

I place my frequently used modules in a subdirectory lib/ and expect to load all modules into main.py by: (refer to Python: how to import from all modules in dir?)

from lib import *

but encounter the issue TypeError: 'module' object is not callable. More specifically, in main.py:

#!/usr/bin/env python

from lib import * # cause: TypeError: 'module' object is not callable
#from lib.DominatingSets import *   # it works

dominatingSets = DominatingSets()

The full exception traceback:

$ python main.py 
Traceback (most recent call last):
  File "main.py", line 6, in <module>
    dominatingSets = DominatingSets()
TypeError: 'module' object is not callable

---

The directories in a tree-like format.

$ tree -P '*.py' .
.
├── __init__.py
├── lib
│   ├── AnalyzeGraph.py
│   ├── AutoVivification.py
│   ├── DominatingSets.py
│   ├── __init__.py
│   ├── Output.py
│   ├── PlotGraph.py
│   ├── ProcessDatasets.py
│   └── ReadGTFS.py
├── main.py

The contents of lib/__init__.py are as follows. (refer to Loading all modules in a folder in Python)

from os.path import dirname, basename, isfile
import glob
modules = glob.glob(dirname(__file__)+"/*.py")
__all__ = [ basename(f)[:-3] for f in modules if isfile(f) and not basename(f).startswith('__')] # exclude __init__.py

Answer 1

This confusion happened, in part, because your module names are the same as the names of the classes you want to load from them. (At least, that's what makes it more confusing.) Your code does correctly load the modules that your classes are in. However, it doesn't load the classes out of those modules, and this is what you actually wanted to do.

Because your class DominatingSets is in the module lib.DominatingSets, its full path from root is lib.DominatingSets.DominatingSets.

from lib import *

in your case will do the same thing as

from lib import DominatingSets
from lib import AnalyzeGraph
# ...

However,

from lib import DominatingSets

is equivalent to

import lib.DominatingSets
DominatingSets = lib.DominatingSets

but lib.DominatingSets is a module (lib/DominatingSets.py), not the class you want.

from lib.DominatingSets import DominatingSets

is equivalent to

import lib.DominatingSets
DominatingSets = lib.DominatingSets.DominatingSets

which is why it works: this is the class you want imported into the name DominatingSets.

If you want to have from lib import * import all the classes in the submodules, you need to import these classes into the lib module. For example, in lib/__init__.py:

from DominatingSets import *
from AnalyzeGraph import *
# ...

While you're making changes, I'd suggest (as others have) using normal Python naming conventions, and have your module names in lowercase: change DominatingSets.py to dominatingsets.py. Then this code would become

from dominatingsets import *
from analyzegraph import *
# ...

Answer 2

Looking at your Traceback, I think your problem might lie here:

Firstly, lets look at an example:

import datetime
d = datetime(2005, 23, 12)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'module' object is not callable

Basically, we've just imported the entire datetime module, and we're trying to call it like a class object within a module. Let's now do:

k = datetime.datetime(2005, 12, 22)
print k
2005-12-22 00:00:00

No problems this time, as we are referencing the datetime object type within the datetime module

If we do:

from datetime import datetime
datetime
<type 'datetime.datetime'>

Again we reach the desired object, as we are importing the datetime class within the datetime module. Also, using *

from datetime import *
d = datetime(2005, 3, 12)

will also work, as you are importing all the classes within the datetime module.

Your code saying:

from lib import * # This imports all MODULES within lib, not the classes
#from lib.DominatingSets import *   # it works because you import the classes within the DominatingSets Module

You could either use from lib.DominatingSets import DominatingSets which should solve your problem, or if you stick to from lib import *, change your code to dominatingsets = DominatingSets.DominatingSets()

Hope this helps!

Answer 3

I learnt a lot from the accepted answer here ... but I still had problems with what to put in the lib/__init__.py file, if this directory lib is not actually included in PYTHONPATH.

I found that in addition to adding the parent directory of lib in the caller file, i.e.

sys.path.append( '.../parent_dir_of_lib' )

I either 1) had to do this in addition in the caller file:

sys.path.append( '.../parent_dir_of_lib/lib' )

Or 2) had to make the lib directory "self-loading", by putting this in its __init__.py:

import sys
from pathlib import Path
parent_dir_str = str( Path(__file__).resolve().parent )
sys.path.append( parent_dir_str )
 
from analyse_graph import *
from auto_vivification import *
...