How do I invoke Python+PYTHONPATH+virtualenv from a Python script?

Question

I would like to write a script to invoke a Python program. I have dabbled in bash and fish, but as the script logic becomes more complex (using Git, etc, etc), I'm leaning increasingly towards using Python. (Both the program and script would be Python 2.7 on Ubuntu 14.04)

However, I have some strong requirements for invoking:

a custom PYTHONPATH must be set (the same every time)
a virtualenv is required (may change based on user input/logic)
program arguments are required (may change based on user input/logic)

The bash equivalent is this:

~$ pew workon <VIRTUALENV>
~$ export PYTHONPATH=</PROGRAM/PATH>
~$ python -m <MODULE.MODULE> --<ARGUMENT> --<ARGUMENT>=<VALUE>

As you can see I use pew to activate my virtualenv rather than virtualenvwrapper or calling the activate script directly. But I'm happy to use any virtualenv library.

Alternatively, is this impossible? (In which case I could write a thin shell wrapper to set the environment, but how do I make the python -m MODULE --ARGUMENT call?)

so, your problem is that you want to call `python -m MODULE --ARGUMENT` from a python script? — jabaldonedo, Jun 20 '14 at 08:48
@jabaldonedo selecting the virtualenv would ideally be part of the logic, and therefore ideally part of the python script - I've updated my question to make this clearer — lofidevops, Jun 20 '14 at 08:53
Have you tried to use subprocesses to invoke those commands from Python? — jabaldonedo, Jun 20 '14 at 08:54
@jabaldonedo no, but thanks for the hint; my concern is how make each call in the same "environment" -- I've been avoiding using python to call a shell script to call python, but is this how one would do it? (I'm curious about a "pure python" solution, but I'm not opposed to pragmatism :) ) — lofidevops, Jun 20 '14 at 09:00
makes me think I could do it with a generic fish wrapper for "call python module X with arguments Y in virtualenv Z", and invoke the wrapper with a python script — lofidevops, Jun 20 '14 at 09:02

score 1 · Answer 1 · answered Jun 20 '14 at 09:06

1

Here is an idea. Define a function that handles the calls to your console and then invoke the function:

import subprocess

''' This functions sends commands to console '''
def command(cmd):
    try:
        p = subprocess.Popen(cmd, shell = True, stdin = subprocess.PIPE, stdout = subprocess.PIPE, stderr = subprocess.STDOUT, close_fds = True)
        stdout = p.communicate()
        return stdout[0]
    except Exception as ex:
        raise Exception(ex)

# now define your variables and make calls

VIRTUALENV = ...
PYTHON_PATH = ...

command("pew workon {}".format(VIRTUALENV))
command("export PYTHONPATH= {}".format(PYTHON_PATH))
command("python -m {} --{} --{}={}".format(...))

answered Jun 20 '14 at 09:06

jabaldonedo

25,822
8
77
77

thanks! I have a shell-based solution thanks to our discussion (see my answer), but I will also try out this pure-python approach too – lofidevops Jun 20 '14 at 09:32
You're welcome. Remember also that you can set/get environment variables within Python: http://stackoverflow.com/questions/5971312/how-to-set-environment-variables-in-python. BTW upvote if you find the answer useful:) – jabaldonedo Jun 20 '14 at 09:37

score 0 · Answer 2 · answered Jun 20 '14 at 09:30

Based on discussion in the comments, it occurred to that I could write a simple, generic shell script for invoking Python + PYTHONPATH + virtualenv + module + arguments, and invoke that simple script from a more complex Python script that determines those values.

#! /usr/bin/fish
# Call a Python module with virtualenv, PYTHONPATH and arguments.
# (TODO: make it more flexible by making arguments optional (plus, learn more fish))

# set variables
set PY_PATH $argv[1]
set PY_VENV $argv[2]
set PY_MODULE $argv[3]
set PY_ARGUMENTS $argv[4..-1]

# invoke Python
set -x PYTHONPATH $PY_PATH
pew in $PY_VENV python -m $PY_MODULE $PY_ARGUMENTS

How do I invoke Python+PYTHONPATH+virtualenv from a Python script?

2 Answers2