3

I have a structure of folders, similar to the one presented below:

main_folder
|
---> sub_folder_1
|    |
|    ---> main_script.py
|    |
|    ---> side_script.py
|
---> sub_folder_2
|    |
|    ---> main_script.py
|    |
|    ---> side_script.py
|
...

i.e. in each sub-folder I have files with the same names. It might happen that for some sub-folders main script imports some objects form the side script, simply by 

import side_script

or

from side_script import *

Obviously, each main script refers to the side script in the same folder it is placed itself.

Now - in a totally different script I want to iterate over subfolders, load the main scripts, use some of their contents and save the results. Since in all sub-folders main files, which I want to import, are named in the same way, I need to closely watch which file I am importing (from which particular sub-folder), so instead of just adding values to the sys.path, I was suggested to use the importlib library and load the modules one by one, using the exec_module() method:

current_path = r'path\to\next\sub\folder'
spec = importlib.util.spec_from_file_location('main_script', os.path.join(current_path, 'main_script.py'))
executed_module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(executed_module)

This works as long as main scripts do not import side scripts. If they do, I get the following error:

ModuleNotFoundError: No module named 'side_module'

Obviously, when I directly run any of the main scripts, there are no problems with that kind of import - they only appear when I import main scripts using exec_module() method. I suspect this is something related to the namespaces (exec_module() executes the module in a new namespace) but still, I don't know how I can make every main script I am executing as a module see the the scripts in the same sub-folder, without altering the main script itself.

Jerry
  • 39
  • 1
  • 3
  • Trying changing the import to `from sub_folder_1.side_script import *` or something similar. Faced similar issue while working with PyCharm. Recommend looking up relative and absolute imports. – MePsyDuck Dec 03 '19 at 12:03
  • I still get an error - `ModuleNotFoundError: No module named 'sub_folder_1'`. But even if it worked, I actually get the content of all the sub-folders from outside and I would definitely prefer to avoid any modifications of them. – Jerry Dec 03 '19 at 12:49
  • You should check your working directory then, the module files are probably not accessible from where you are executing the runner script. Do check this https://stackoverflow.com/questions/56603077/modulenotfounderror-when-using-importlib-import-module – MePsyDuck Dec 03 '19 at 15:44

3 Answers3

1

As long as you got you __init__.py inside each sub_folder you can simply add the folder to the Python path. The Python path is what will be searched when you use import statements. E.g.:

import sys
sys.path.append(str(/full/path/to/sub_folder/))

I have not yet found a more elegant way to use importlib for this job or specify that the folder containing the module you're calling exec_module on is automaticaly added to the Python path. However, since you're manipulating the Python path this means that it can lead to unexpected behaviour when you return to the module that started the execution or start another module. Thus check this answer which shows an example implementation of how to reset the python path to it's original state after every call.

omni
  • 4,104
  • 8
  • 48
  • 67
-1

you need create init.py in all folders to define the modules

main_folder
|    |
---> __init__.py   
|    |
---> sub_folder_1
|    |
|    ---> __init__.py   
|    |
|    ---> main_script.py
|    |
|    ---> side_script.py
|
---> sub_folder_2
|    |
|    ---> __init__.py    
|    |
|    ---> main_script.py
|    |
|    ---> side_script.py
|
...
skomlev
  • 27
  • 3
  • This is only valid for pre Python 3.3 versions. v3.3 onwards you don't need `__init__.py` for python to treat directory as module. – MePsyDuck Dec 03 '19 at 12:05
-1

There is a similar question here, however the suggested solution are not satisfactory as they imply to change sys.path and I think what you want is precisely to import a module as if it was in your working folder.

What I propose is to add your module in sys.modules before executing it:

sys.modules[executed_module.__name__] = executed_module # add this line
spec.loader.exec_module(executed_module) # before executing the module

Now you should be able to import siblings module in the executed module with:

from . import sibling
Antonin Rousset
  • 111
  • 4