Python multiprocessing.Process to use virtualenv

Question

I am running a program using virtualenv. But the multiprocessing.Process spawned here uses system python by default. How do I force it to use virtualenv python.

import os
from multiprocessing import Process

def function1():
    # do_something_here
    p = Process(func2(), args=(param,))
    p.start()
    return something

def func2(param):
    os.system("which python")

Here it prints "/usr/bin/python". But I need it to use virtualenv python instead.

@georgexsh Yes it is. Actually these functions are part of a flask app. — Kumaran, Oct 27 '17 at 15:34
I tested a fixed version of the above program (for example you do not have `func2()` as Process parameter but `func2`, and some other fixes), and it indeed returns the virtualenv version. Try adding `os.system("which python")` to the main program just before you spawn your process and see what happens there. — Hannu, Oct 27 '17 at 15:55
@georgexsh I was running it using sudo. By default if you use sudo then it will use system python. So, I have used "sudo venv/bin/python main.py" to run the program. Even though I am using venv's python here it returns "/usr/bin/python" for "os.system('which python')". I don't understand this behaviour. — Kumaran, Nov 01 '17 at 06:34
@Kumaran please take caution of the answer you accepted, it is inaccurate. — georgexsh, Nov 04 '17 at 07:03
@Kumaran PYTHONPATH/PATH/sudo all of them have nothing to do with your issue, I updated my answer accordingly. — georgexsh, Nov 06 '17 at 19:43

georgexsh · Accepted Answer · 2017-11-06T19:50:21.013

With sudo venv/bin/python, you effectively activated virtualenv by using python executable in virtualenv directly.

multiprocessing.Process spawn child process with fork(), without exec(), it uses exactly the same python executable as the parent process.

You could confirm python executable in-use by:

>>> import sys
>>> print(sys.executable)
/Users/georgexsh/workspace/tmp/venv/bin/python
>>> print(sys.exec_prefix)
/Users/georgexsh/workspace/tmp/venv/bin/..

Do not use which to determine running python executable path. which, as a Bash command, searches each element of $PATH, for a directory containing an executable file named "python", as you use virtualenv's python directly, not by run its shell activate script first, $PATH not get patched with virtualenv's, as a result, shell command which python would output path of the system python executable.

In fact, at python layer, $PATH is irrelevant, patching $PATH is for the convenience at the Bash layer, to invoke python executable in the virtualenv path by just typing "python", rather than typing full path, What matters most is which python executable is invoked, but not how it is get invoked.

This is correct. I tried it. "sys.executable" prints venv's python path but "which python" prints system python. Thanks @georgexsh — Kumaran, Nov 08 '17 at 05:42

score 1 · Answer 2 · answered Nov 01 '17 at 09:58

Your problem is here (copied your comment):

@georgexsh I was running it using sudo. By default if you use sudo then it will use system python. So, I have used "sudo venv/bin/python main.py" to run the program. Even though I am using venv's python here it returns "/usr/bin/python" for "os.system('which python')". I don't understand this behaviour

Basically, what you explain here is something where your virtualenv is not active.

When you activate a virtualenv (. venv/bin/activate), the activation script will change your environment so that your PYTHONPATH is correct and Python executable is searched (and found) first in the virtual env directory. This is what virtualenv does.

By just executing the Python binary from virtualenv directories, your environment is not set for the virtual environment, so any subsequent calls to Python use your default path - as virtualenv is not there to override it.

When you execute sudo, a new process/shell is created and it does not inherit your virtual environment. You might be able to use sudo -E to pass environment but it depends on your sudo. The bulletproof version that should work in every environment is to execute a shell that first activates virtualenv and then executes your script. Something like this:

sudo -- bash  -c ". /home/test/mytest/bin/activate; which python"

This executes a bash shell as root, then activates the virtual environment and finally tells you which python it uses. Just modify the above command with your virtual environment path and it might even work.

If your system is shared, just keep in mind that this is a horrible thing to allow your regular users do from security perspective. If you create a passwordless sudo for your regular users to do this, it would give them root access with little tweaking. If it is your own system and the requirement is the knowledge of root password anyway, it does not matter.

`PYTHONPATH` has nothing to do here, and virtualenv is *active* here, `Process` use virtualenv python too, your other statements are plainly false too... — georgexsh, Nov 04 '17 at 06:59
As the comments state, OP may have virtualenv initially active, but then he uses sudo to execute Python and his script, which replaces the venv PATH with his or system's default. — Hannu, Nov 04 '17 at 23:31
`sudo venv/bin/python` *effectively* activated virtualenv by using python executable in virtualenv. — georgexsh, Nov 05 '17 at 04:52

Python multiprocessing.Process to use virtualenv

2 Answers2