Load all modules (files) in directory without prefixes

Question

I am using python 2.7. I have a folder with a number of .py files in it that define functions, load objects, etc. that I would like to use in my main script. I want to accomplish the following two objectives:

1. Load all objects from all .py files in this directory, without knowing the file names in advance.
2. Access these objects in my main script without prefixing, e.g. without needing to use filename.function_name()

I understand that this does not conform to accepted best practices for python modules. Nevertheless:

• I am writing code purely for my own use. It will never be shared with or used by others.

• In the course of my development work, I frequently change the names of files, move object definitions from one file to another, etc. Thus, it is cumbersome to need to go to my main script and change the names of the function name prefixes each time I do this.

• I am an adult. I understand the concept of risks from name conflicts. I employ my own naming conventions with functions and objects I create to ensure they won't conflict with other names.

My first shot at this was to just loop through the files in the directory using os.listdir() and then call execfile() on those. When this did not work, I reviewed the responses here: Loading all modules in a folder in Python . I found many helpful things, but none get me quite where I want. Specifically, if include in my __init__.py file the response here:

from os.path import dirname, basename, isfile
import glob
modules = glob.glob(dirname(__file__)+"/*.py")
__all__ = [ basename(f)[:-3] for f in modules if isfile(f)]

and then use in my main script:

os.chdir("/path/to/dir/") # folder that contains `Module_Dir`
from Module_Dir import *

then I can gain access to all of the objects defined in the files in my directory without needing to know the names of those files ahead of time (thus satisfying #1 from my objectives). But, this still requires me to call upon those functions and objects using filename.function_name(), etc. Likewise, if I include in my __init__.py file explicitly:

from filename1 import *
from filename2 import *
etc.

Then I can use in my main script the same from Module_Dir import * as above. Now, I get access to my objects without the prefixes, but it requires me to specify explicitly the names of the files in __init__.py.

Is there a solution that can combine these two, thus accomplishing both of my objectives? I also tried (as suggested here, for instance, including the following in __init__.py:

import os
for module in os.listdir(os.path.dirname(__file__)):
    if module == '__init__.py' or module[-3:] != '.py':
        continue
    __import__(module[:-3], locals(), globals())
del module

But again, this still required the name prefixing. I tried to see if there were optional arguments or modifications to how I use __import__ here, or applications using python 2.7's importlib, but did not make progress on either front.

Answer 1

Ok, here is the solution that I came up with based on the suggestion from @meetaig. I'd be delighted though to accept and upvote any other ideas or suggested implementations for this situation:

import os

def Load_from_Script_Dir(DirPath):
    os.chdir(os.path.dirname(os.path.dirname(DirPath))) ## so that the from .. import syntax later properly finds the directory.
    with open(DirPath + "__init__.py", 'w') as f:
        Scripts = [FileName for FileName in os.listdir(DirPath) if FileName[-3:] == '.py' and '__init__' not in FileName]
        for Script in Scripts: 
            f.write('from %s import *\n'  % Script.replace('.py',''))


# Script_Dir = "/path/to/Script_Dir/"  ## specify location of Script_Dir manually
Script_Dir = os.path.dirname(os.path.abspath(__file__)) + "/Script_Dir/" ## assume Script_Dir is in same path as main.py that is calling this.  Note: won't work when used in REPL.
Load_from_Script_Dir(Script_Dir)
from Script_Dir import *

Notes:

1. this will also require that the .py files in the Script_Dir conform to naming conventions for python objects, i.e. they cannot start with numbers, cannot have spaces, etc.

2. the .py files in Script_Dir here that get loaded won't have access to objects defined in the main script (even if they are defined before calling from Script_Dir import *). Likewise, they won't have access to objects defined in other scripts within Script_Dir. Thus, it may still be necessary to use a syntax such as from filename import objectname within those other .py files. This is unfortunate and partially defeats some of the convenience I was hoping to achieve, but it at least is a modest improvement over the status quo.

Answer 2

Here is another solution, based on the same general principle as in my other implementation (which is inspired by the suggestion from @meetaig in comments). This one bypasses attempts to use the python module framework, however, and instead develops a workaround to be able to use execfile(). It seems much preferable for my purposes because:

1. If one support script defines functions that use functions or objects defined in other scripts, there is no need to add an extra set of from filename import ... to it (which as I noted in my other answer, defeated a lot of the convenience I had hoped to gain). Instead, just as would happen if the scripts were all together in a single big file, the functions / objects only need be defined by the time a given function is called.

2. Likewise, the support scripts can make use of objects in the main script that are created prior to the support scripts being run.

3. This avoids creating a bunch of .pyc files that just go and clutter up my directory.

4. This similarly bypasses the need (from my other solution) of making sure to navigate the interpreter's working directory to the proper location in order to ensure success of the from ... import ... statement in the main script.

.

import os

Script_List_Name = "Script_List.py"

def Load_from_Script_Dir(DirPath):
    with open(DirPath + Script_List_Name, 'w') as f:
        Scripts = [FileName for FileName in os.listdir(DirPath) if FileName[-3:] == '.py' and FileName != Script_List_Name]
        for Script in Scripts: 
            f.write('execfile("%s")\n'  % (DirPath + Script))

Script_Dir = "/path/to/Script_Dir/"

Load_from_Script_Dir(Script_Dir)
execfile(Script_Dir + Script_List_Name)

Alternatively, if it isn't necessary to do the loading inside a function, the following loop also works just fine if included in the main script (I was thrown off initially, because when I tried to build a function to perform this loop, I think it only loaded the objects into the scope of that function, and thus they weren't accessible elsewhere, an operation which is different from other languages like Julia, where the same function would successfully put the objects into a scope outside the function):

import os
Script_Dir = "/path/to/Script_Dir/"
for FileName in os.listdir(Script_Dir):
    if FileName[-3:] == '.py':
        print "Loading %s" %FileName
        execfile(Script_Dir + FileName)

Answer 3

Using Path

"""
Imports all symbols from all modules in the package
"""
from pathlib import Path
from importlib import import_module


for module_path in Path(__file__).parent.glob('*.py'):
    if module_path.is_file() and not module_path.stem.startswith('_'):
        module = import_module(f'.{module_path.stem}', package=__package__)
        symbols = [symbol for symbol in module.__dict__ if not symbol.startswith("_")]
        globals().update({symbol: getattr(module, symbol) for symbol in symbols})