Best practice for handling path/executables in project scripts in Python (e.g. something like Django's manage.py, or fabric)

Question

I do a lot of work on different projects (I'm a scientist) in a fairly standardised directory structure. e.g.:

project
   /analyses/
   /lib
   /doc
   /results
   /bin

I put all my various utility scripts in /bin/ because cleanliness is next to godliness. However, I have to hard code paths (e.g. ../../x/y/z) and then I have to run things within ./bin/ or they break.

I've used Django and that has /manage.py which runs various django-things and automatically handles the path. I've also used fabric to run various user defined functions.

Question: How do I do something similar? and what's the best way? I can easily write something in /manage.py to inject the root dir into sys.path etc, but then I'd like to be able to do "./manage.py foo" which would run /bin/foo.py. Or is it possible to get fabric to call executables from a certain directory?

Basically - I want something easy and low maintenance. I want to be able to drop an executable script/file/whatever into ./bin/ and not have to deal with path issues or import issues.

What is the best way to do this?

Answer 1

Keep Execution at TLD

In general, try to keep your runtime at top-level. This will straighten out your imports tremendously.

If you have to do a lot of import addressing with relative imports, there's probably a better way.

Modifying The Path

Other poster's have mentioned the PYTHONPATH. That's a great way to do it permanently in your shell.

If you don't want to/aren't able to manipulate the PYTHONPATH project path directly you can use sys.path to get yourself out of relative import hell.

Using sys.path.append

sys.path is just a list internally. You can append to it to add stuff to into your path.

Say I'm in /bin and there's a library markdown in lib/. You can append a relative paths with sys.path to import what you want.

import sys
sys.path.append('../lib')
import markdown


print markdown.markdown("""

Hello world!
------------

""")

Word to the wise: Don't get too crazy with your sys.path additions. Keep your schema simple to avoid yourself a lot confusion.

Overly eager imports can sometimes lead to cases where a python module needs to import itself, at which point execution will halt!

Using Packages and __init__.py

Another great trick is creating python packages by adding __init__.py files. __init__.py gets loaded before any other modules in the directory, so it's a great way to add imports across the entire directory. This makes it an ideal spot to add sys.path hackery.

You don't even need to necessarily add anything to the file. It's sufficient to just do touch __init__.py at the console to make a directory a package.

See this SO post for a more concrete example.

Answer 2

In a shell script that you source (not run) in your current shell you set the following environment variables:

PATH=$PATH:$PROJECTDIR/bin
PYTHONPATH=$PROJECTDIR/lib

Then you put your Python modules and package tree in your projects ./lib directory. Python automatically adds the PYTHONPATH environment variable to sys.path.

Then you can run any top-level script from the shell without specifying the path, and any imports from your library modules are looked for in the lib directory.

I recommend very simple top-level scripts, such as:

#!/usr/bin/python

import sys
import mytool

mytool.main(sys.argv)

Then you never have to change that, you just edit the module code, and also benefit from the byte-code caching.

Answer 3

You can easily achieve your goals by creating a mini package that hosts each one of your projects. Use paste scripts to create a simple project skeleton. And to make it executable, just install it via setup.py develop. Now your bin scripts just need to import the entry point to this package and execute it.