How to maintain python application with dependencies, including my own custom libs?

Question

I'm using Python to develop few company-specific applications. There is a custom shared module ("library") that describes some data and algorithms and there are dozens of Python scripts that work with this library. There's quite a lot of these files, so they are organized in subfolders

myproject
    apps
        main_apps
            app1.py
            app2.py
            ...
        utils
            util1.py
            util2.py
            ...
    library
        __init__.py
        submodule1
            __init__.py
            file1.py
            ...
        submodule2
            ...

Users want to run these scripts by simply going, say, to myproject\utils and launching "py util2.py some_params". Many of these users are developers, so quite often they want to edit a library and immediately re-run scripts with updated code. There are also some 3rd party libraries used by this project and I want to make sure that everyone is using the same versions of these libs.

Now, there are two key problems I encountered:

1. how to reference (library) from (apps)?
2. how to manage 3rd party dependencies?

The first problem is well-familiar to many Python developers and was asked on SO for many times: it's quite difficult to instruct Python to import package from "....\library". I tested several different approaches, but it seems that python is reluctant to search for packages anywhere, but in standard libraries locations or the folder of the script itself.

• Relative import doesn't work since script is not a part of a library (and even if it was, this still doesn't work when script is executed directly unless it's placed in the "root" project folder which I'd like to avoid)
• Placing .pth file (as one might think from reading this document) to script folder apparently doesn't have any effect

Of course direct meddling with sys.path work, but boilerplate code like this one in each and every one of the script files looks quite terrible

import sys, os.path
here = os.path.dirname(os.path.realpath(__file__))
module_root = os.path.abspath(os.path.join(here, '../..'))
sys.path.append(python_root)
import my_library

I realize that this happens because Python wants my library to be properly "installed" and that's indeed would be the only right way to go had this library was developed separately from the scripts that use it. But unfortunately it's not the case and I think that re-doing "installation" of library each time it's changed is going to be quite inconvenient and prone to errors.

The second problem is straightforward. Someone adds a new 3rd party module to our app/lib and everyone else start seeing import problems once they update their apps. Several branches of development, different moments when user does pip install, few rollbacks - and everyone eventually ends using different versions of 3rd party modules. In my case things are additionally complicated by the fact that many devs work a lot with older Python 2.x code while I'd like to move on to Python 3.x

While looking for a possible solution for my problems, I found a truly excellent virtual environments feature in Python. Things looked quite bright:

1. Create a venv for myproject
2. Distribute a Requirements.txt file as part of app and provide a script that populates venv accordingly
3. Symlink my own library to venv site_packages folder so it'll be always detected by Python

This solution looked quite natural & robust. I'm explicitly setting my own environment for my project and place whatever I need into this venv, including my own lib that I can still edit on the fly. And it indeed work. But calling activate.bat to make this python environment active and another batch file to deactivate it is a mess, especially on Windows platform. Boilerplate code that is editing sys.path looks terrible, but at least it doesn't interfere with UX like this potential fix do.

So there's a question that I want to ask.

1. Is there a way to bind particular python venv to particular folders so python launcher will automatically use this venv for scripts from these folders?
2. Is there a better alternative way to handle this situation that I'm missing?

Environment for my project is Python 3.6 running on Windows 10.

Answer 1

I think that I finally found a reasonable answer. It's enough to just add shebang line pointing to python interpreter in venv, e.g.

#!../../venv/Scripts/python

The full project structure will look like this

myproject
    apps
        main_apps
            app1.py (with shebang)
            app2.py (with shebang)
            ...
        utils
            util1.py (with shebang)
            util2.py (with shebang)
            ...
    library
        __init__.py
        submodule1
            __init__.py
            file1.py
            ...
        submodule2
            ...
    venv
        (python interpreter, 3rd party modules)
        (symlink to library)
    requirements.txt
    init_environment.bat

and things work like this:

1. venv is a virtual python environment with everything that project needs
2. init_environment.bat is a script that populates venv according to requirements.txt and places a symlink to my library into venv site-modules
3. all scripts start with shebang line pointing (with relative path) to venv interpreter

There's a full custom environment with all the libs including my own and scripts that use it will all have very natural imports. Python launcher will also automatically pick Python 3.6 as interpreter & load the relevant modules whenever any user-facing script in my project is launched from console or windows explorer.

Cons:

1. Relative shebang won't work if a script is called from other folder
2. User will still have to manually run init_environment.bat to update virtual environment according to requirements.txt
3. init_environment scrip on Windows require elevated privileges to make a symlink (but hopefully that strange MS decision will be fixed with upcoming Win10 update in April'17)

However I can live with these limitations. Hope that this will help others looking for similar problems.

Would be still nice to still hear other options (as answers) and critics (as comments) too.