0

Is there a way (using only python. i.e.: without a bash script nor another language code) to call a specific function in every script inside a folder without needing to import all of them explicitly.

For example, let's say that this is my structure:

main.py
modules/
    module1.py
    module2.py
    module3.py
    module4.py

and every moduleX.py has this code:

import os

def generic_function(caller):
    print('{} was called by {}'.format(os.path.basename(__file__), caller))

def internal_function():
    print('ERROR: Someone called an internal function')

while main.py has this code:

import modules
import os

for module in modules.some_magic_function():
    module.generic_function(os.path.basename(__file__))

So if I run main.py, I should get this output:

module1.py was called by main.py
module2.py was called by main.py
module3.py was called by main.py
module4.py was called by main.py

*Please note that internal_function() shouldn't be called (unlike this question). Also, I don't want to declare explicitly every module file even on a __init__.py

By the way, I don't mind to use classes for this. In fact it could be even better.

JuanP. Zuniga
  • 65
  • 3
  • 13
  • 1
    Can you clarify what you mean by "without needing to import all of them"? Do you mean without *explicitly* importing them, e.g. ``import module1``, or do you mean without importing them *at all*, e.g. to only implicitly import those that have ``generic_function`` defined? – MisterMiyagi Nov 09 '20 at 13:37
  • @MisterMiyagi By "without needing to import all of them" I mean to not needing to do the `import module1, module2, module3, module4, module5` (i.e. importing them explicitly), but I need to have at least a kind of control over the modules (for example, if `generic_function()` returns True for some module, then call another function for the same module) – JuanP. Zuniga Nov 09 '20 at 14:04

2 Answers2

1

You can use exec or eval to do that. So it would go roughly this way (for exec):

def magic_execute():
    import os
    import glob
    for pyfl in glob.glob(os.path(MYPATH, '*.py'):
        with open(pyfl, 'rt') as fh:
            pycode = fh.read()
            pycode += '\ngeneric_function({})'.format(__file__)
            exec(pycode)

The assumption here is that you are not going to import the modules at all.

Please note, that there are numerous security issues related to using exec in such a non-restricted manner. You can increase security a bit.

sophros
  • 14,672
  • 11
  • 46
  • 75
  • Sorry for not being clear about how I want to import the modules. `eval` could be a good choice because some functions will return some values (mainly booleans), but I'm worried about reading and calling every module could generate a memory leak or something like that as my original main.py runs indefinitely (for sake of the question simplicity I didn't tell this). Is there a way to have something like your proposed code but importing the modules only once for each one, then referencing them from a list or something like that? – JuanP. Zuniga Nov 09 '20 at 14:18
  • This is an unfounded worry - the modules are going to be imported only once per loop and garbage-collected after execution. There is no worry of memory leak with this approach. – sophros Nov 09 '20 at 14:22
  • Nice to know (I had a hunch but I wasn't sure). I have another question: what about calling two functions for the same module but depending of the first one's value. For example: if `generic_function()` returns `true`, then call `internal_function()` (for the same module) and break the loop. Otherwise, go to the next module and repeat the process. – JuanP. Zuniga Nov 09 '20 at 14:30
  • You can add a piece of code to execute an `if` statement (similarly as I added execution of your `generic_function`. BTW, this should really be a different question. – sophros Nov 09 '20 at 14:31
0

While sophros' approach is quickly and enough for implicitly importing the modules, you could have issues related to controlling every module or with complex calls (like having conditions for each calls). So I went with another approeach:

First I created a class with the function(s) (now methods) declared. With this I can avoid checking if the method exists as I can use the default one if I didn't declare it:

# main.py
class BaseModule:
    def __init__(self):
        # Any code
    
    def generic_function(self, caller):
        # This could be a Print (or default return value) or an Exception
        raise Exception('generic_function wasn\'t overridden or it was used with super')

Then I created another class that extends the BaseModule. Sadly I wasn't able to get a good way for checking inherence without knowing the name of the child class so I used the same name for every module:

# modules/moduleX.py
from main import BaseModule

class GenericModule(BaseModule):
    def __init__(self):
        BaseModule.__init__(self)
        # Any code
    
    def generic_function(self, caller):
        print('{} was called by {}'.format(os.path.basename(__file__), caller))

Finally, in my main.py, I used the importlib for importing the modules dynamically and saving an instance for each one, so I can use them later (for sake of simplicity I didn't save them in the following code, but it's easy as using a list and appending every instance on it):

# main.py
import importlib
import os

if __name__ == '__main__':
    relPath = 'modules' # This has to be relative to the working directory

    for pyFile in os.listdir('./' + relPath):
        # just load python (.py) files except for __init__.py or similars
        if pyFile.endswith('.py') and not pyFile.startswith('__'):
            # each module has to be loaded with dots instead of slashes in the path and without the extension. Also, modules folder must have a __init___.py file
            module = importlib.import_module('{}.{}'.format(relPath, pyFile[:-3]))
            # we have to test if there is actually a class defined in the module. This was extracted from [1]
            try:
                moduleInstance = module.GenericModule(self)
                moduleInstance.generic_function(os.path.basename(__file__)) # You can actually do whatever you want here. You can save the moduleInstance in a list and call the function (method) later, or save its return value.
            except (AttributeError) as e:
                # NOTE: This will be fired if there is ANY AttributeError exception, including those that are related to a typo, so you should print or raise something here for diagnosting
                print('WARN:', pyFile, 'doesn\'t has GenericModule class or there was a typo in its content')

References:

[1] Check for class existence

[2] Import module dynamically

[3] Method Overriding in Python

JuanP. Zuniga
  • 65
  • 3
  • 13