global initialize function and global constants in Python without using sys.path.append()

Question

My use case is the following:

• several scripts that can be run individually
• one 'mother' script that can run all the scripts
• each script has global variables it uses (paths to specific locations, constants used by all scripts)
• some of these global variables are inputted in a settings.yaml file which gets dynamically loaded with yaml.load()
• several people are running the system, we want to avoid environment variables or any kind of low-level path settings

The use cases were satisfied with an

import settings

at the beginning of each export module.

The settings.py script is our 'initialize' script that loads and create variables used for all the modules, e.g.:

import os
import yaml
from utils import create_dir

input_f = open("settings.yml")
settings = yaml.load(input_f.read())
input_f.close()

TOHELLO = settings['TOHELLO']
TOBONJOUR = settings['TOBONJOUR']

#check for needed directories and create if necessary
create_dir(TOHELLO)
create_dir(TOBONJOUR)

The 'mother module' export_all.py uses Subprocess.Popen() to run all the scripts sequentially.

Originally all files were in the root:

+--settings.py
+--settings.yml
+--export_foo1.py
+--export_foo2.py
+--export_foo3.py
+--export_foo4.py
+--export_bar1.py
+--export_bar2.py
+--export_bar3.py
+--export_bar4.py
+--export_all.py

With time, though, this became messy and we decided to restructure it:

+--settings.py
+--settings.yml
+--Foo
+  +--export_foo1.py
+  +--export_foo2.py
+  +--export_foo3.py
+  +--export_foo4.py
+--Bar
+  +--export_bar1.py
+  +--export_bar2.py
+  +--export_bar3.py
+  +--export_bar4.py
+--export_all.py

And then all hell broke loose.

We discovered that one can only use 'import' for modules that are either underneath the module or specifically within sys.path.

The solution we've come up with involves inserting this beauty at the top of all of our scripts:

PACKAGE_PARENT = os.path.join('..','..')
SCRIPT_DIR = os.path.dirname(os.path.realpath(os.path.join(os.getcwd(), os.path.expanduser(__file__))))
sys.path.append(os.path.normpath(os.path.join(SCRIPT_DIR, PACKAGE_PARENT)))

but it just seems like overkill to us.

We tried

from settings import functionname

but that doesn't run the code that generates the constants from the yaml file

Also, we tried squeezing the imports in __init.py__ in each directory, but then the Subprocess.Popen() didn't work.

We're kind of missing a pythonesque way to have some global constants, and an initialiazing script that runs only once (even when the 'mother' script is running).

Is it us, or is python just weird and ugly here?

Answer 1

You need to turn your directories into modules: https://docs.python.org/2/tutorial/modules.html#packages

to allow you use a structure like this:

from foo import export_foo1

or to use:

from ..bar import export_bar1

in case you are inside a subfolder

https://docs.python.org/2/tutorial/modules.html#intra-package-references

Answer 2

I am not sure what exactly the problem is, but here are several options you can choose from:

1. The best approach is to reference your scripts as if they are submodules of a package. In this case, everything should work if you run all scripts from the directory where export_all.py lives. The only change is that you should run them as modules, and not as scripts:

   $ python export_all.py
   

   but

   $ python -m Foo.export_foo
   

2. You can use PYTHONPATH environment variable to pass additional paths for sys.path to your script. The variable will be inherited by subprocesses.

3. Assuming you run your scripts in a virtual environment, you can put a .pth file in your site-packages directory with absolute paths of the Foo and Bar directories. The paths will be added to sys.path each time a script is started with python without -S switch. You can read about .pth files in the documentation for standard site module.