46

Let's say I have 2 different versions of my app installed in 2 different virtualenvironments. myapp v1.0 and myapp v2.0.

Now I would like to compare those. The comparison is written in python itself. What would be the best way to do that? Let's assume I can run them separately and both write an output file, which I can compare later.

One way to do that would be to write a bash script (that's what I have currently). I activate one virtualenv, run myapp v1.0, activate another virtualenv, run myapp v2.0. Later run a comparison module on those files. But I would like to add some more dynamics there (take some optional arguments etc) which would be easier with python.

Edit:

Currently I have something like that (a bash script):

source virtualenv1/bin/activate
python my_script.py
deactivate

source virtualenv2/bin/activate
python my_other_script.py
deactivate

python my_comparison_script.py

instead, I'd like to do only:

python my_comparison_script.py

and my scripts would be run inside this.

Ago
  • 1,101
  • 3
  • 12
  • 22

3 Answers3

74

The accepted answer does not address the problem of 'activating' a virtualenv in a subprocess.

If you start your application with a call to the python executable, like in your example it is actually very simple: you only have to explicitly point to the executable in the virtualenv.

import subprocess

subprocess.Popen(["virtualenv1/bin/python", "my_script.py"])
subprocess.Popen(["virtualenv2/bin/python", "my_other_script.py"])

will start the processes in the respective virtualenvs.

Important

To address the concerns voiced in the comments:

If you want to run a subprocess and be sure to use the same interpreter that the current process is running in you have to use sys.executable. Also available: sys.exec_prefix to access the site-specific directory prefix where the platform-dependent Python files are installed.

If you want a much more in depth discussion of this topic, have a look at this pull request.

Oliver Bestwalter
  • 5,219
  • 33
  • 47
  • 23
    Actually, this is not enough. Activating a virtualenv also changes some environment variables, calling specific virutalenv's python doesn't do that. As a result, wrong libraries will be loaded. – Dima Tisnek Dec 08 '14 at 09:47
  • 2
    @qarma: I disagree - this approach works and I am using it in my own production code. I am [not the only one recommending it](http://stackoverflow.com/a/14792407/2626627), either. I would be helpful if you concretize the environment variables you are referring to and provide a test case that shows under wich circumstances wrong libraries would be loaded. – Oliver Bestwalter Dec 08 '14 at 15:18
  • 2
    Yes I've seen that recommendation on virtualenv's site too. There are at least two cases where running with virtualenv linked python interpreter directly is different from using activate: when `PYTHONHOME` is set and when target program uses `command` or `Popen` to run another Python program. The latter if arguably murky. – Dima Tisnek Dec 09 '14 at 08:57
  • 12
    I just tried the second scenario and that's true: calling subprocess from inside my_script.py would be exectuted with the system Python again. Awkward. – Oliver Bestwalter Dec 09 '14 at 11:15
  • 2
    On windows. I need to "set PYTHONPATH= & workon virtualenv1" before calling python or the script. Otherwise, the python interpreter PYTHONPATH still points back to the system libraries. – jonincanada Aug 20 '18 at 21:36
  • Sorry for hijacking this but how does tox handle this? – Charlie Clark Aug 22 '18 at 13:20
14

I think virtualenv documentation explains it nicely.

TL;DR

Runnig python venv binary directly is NOT the same as activating venv. You also have to change PATH and VIRTUAL_ENV variables accordingly (look at os.environ)

Source

$ source /path/to/ENV/bin/activate

This will change your $PATH so its first entry is the virtualenv’s bin/ directory. (You have to use source because it changes your shell environment in-place.) This is all it does; it’s purely a convenience.

If you directly run a script or the python interpreter from the virtualenv’s bin/ directory (e.g. path/to/ENV/bin/pip or /path/to/ENV/bin/python-script.py) then sys.path will automatically be set to use the Python libraries associated with the virtualenv. But, unlike the activation scripts, the environment variables PATH and VIRTUAL_ENV will not be modified. This means that if your Python script uses e.g. subprocess to run another Python script (e.g. via a !/usr/bin/env python shebang line) the second script may not be executed with the same Python binary as the first nor have the same libraries available to it. To avoid this happening your first script will need to modify the environment variables in the same manner as the activation scripts, before the second script is executed.

Nathaniel Jones
  • 939
  • 1
  • 14
  • 25
eleksis
  • 141
  • 1
  • 4
12

A simple option would be to run a series of commands with subprocess as follows (note that shell=True is risky and should only be used if you can control the input).

import subprocess
    
cmd = 'source activate my_virtualenv; python my_script.py'
subprocess.call(cmd, shell=True, executable='/bin/bash')

And repeat as needed.

MANORIT CHAWDHRY
  • 403
  • 3
  • 11
fiskhe
  • 137
  • 1
  • 4
  • 2
    `Source` won't be available inside subprocess because subprocess uses /bin/sh. This will give the error: "/bin/sh: 1: source: not found" – emschorsch Feb 24 '20 at 23:17
  • 4
    Use `.` instead of `source`, which is the portable equivalent – remram Mar 02 '20 at 19:53
  • Perhaps highight that the effects of `source` will disappear when `subprocess.call` finishes. The main meat here is running your main Python script in the same subprocess as the `source` command, which is often unattractive for other reasons. (Maybe instead then run a shell script which does the `source` and _then_ runs your entire Python script.) – tripleee Jun 17 '22 at 05:20