Accessing common properties from multiple python modules

Question

I'm not entirely sure if the title is appropriate for this question but here's what I am trying to achieve: I have a directory structure like this:

models/
   modelA/
      src/
         modelA.py
      __init__.py
   modelB/
      src/
          modelB.py
      __init__.py
scripts/
   sandbox.py

The __init__.py file within each model folder contains the following code:

from .src.modelA import modelA as Model

MODEL_NAME = "modelA"
MODEL_VERSION = "1.0.0"

INPUT_FEATURES_TYPES = [ ("feature1", "1.0.0"), ("feature2", "1.0.0") ]
INPUT_TYPE = "images"

Now from within sandbox.py, I want to be able to create an array containing the list of models and their properties by parsing each of the __init__.py files. The final result I want to achieve is something like this:

[{
    name : "modelA",
    version: "1.0.0",
    inputFeat : [("feature1", "1.0.0"), ("feature2", "1.0.0")],
    inputType : "images"
 }, 
 {
    name : "modelB",
    version: "1.0.0",
    inputFeat : [("feature1", "1.0.0"), ("feature2", "1.0.0")],
    inputType : "text"
 },
 ...
]

Is there any easy way to do this? or do I HAVE to create and update a static file that lists all these details? My intention is that I can keep adding new models in the future and still be able to easily extract this list whenever needed.

gohu · Accepted Answer · 2020-05-20T09:21:31.960

1

You can use pkgutil to iterate over the submodules:

import pkgutil
import importlib

for importer, modname, ispkg in pkgutil.iter_modules('/path/to/models', '.src.'):
   module = importlib.import_module(modname)
   from module import MODEL_NAME, MODEL_VERSION, INPUT_FEATURES_TYPES, INPUT_TYPE

Then you can create your dictionary.

I haven't tested the previous code but it works on one my own packages using the module name:

pkgutil.iter_modules(module.__path__, module.__name__ + '.')

edited May 20 '20 at 09:21

answered May 20 '20 at 08:16

gohu

151
1
9

This has almost solved my problem but I don't understand this behaviour: `for importer, modname, ispkg in pkgutil.iter_modules(package.__path__):` `if (ispkg):` `print(prefix + modname)` `model = __import__(prefix + modname)` `print(model)` The output is: `aiscripts.models.geom_resnet` `` `aiscripts.models.some_model` `` It PRINTS the correct submodule but imports only the top level module. So I cannot access the properties of the submodule! – Pawan Bhandarkar May 20 '20 at 09:04
Have you tried adding the prefix to the iter_module function like I did: `pkgutil.iter_modules(module.__path__, module.__name__ + '.')` – gohu May 20 '20 at 09:08
Yes I did. The same result. The modname is correct but only the top level module is actually imported. – Pawan Bhandarkar May 20 '20 at 09:12
Try `model = __import__(prefix+modname, fromlist='dummy')` – gohu May 20 '20 at 09:12
Or you might want to use: `importlib.import_module('prefix+modname')` – gohu May 20 '20 at 09:19
okay so I checked the docs and the behaviour of `__import__` is to infact import the top level module. So I had to use `import_module()` function and that solved my problem. https://docs.python.org/3/library/importlib.html#importlib.import_module – Pawan Bhandarkar May 21 '20 at 04:42
thanks for the help! Marked as the accepted answer! – Pawan Bhandarkar May 25 '20 at 02:26

Accessing common properties from multiple python modules

1 Answers1