How to call pip from a python script and make it install locally to that script?

Question

We have a python script in our source code repository which I maintain. Let's imagine it is at location

scripts/python/make_salad/make_salad.py

It is checked into the repository as is. Users of the script want to just click on the script in windows explorer and off they go. They refuse to use the command line. However the script also depends on many external packages that I have to install. I've used the following trick to install any packages required the first time the user runs the script. It goes like

def install(package):
    # This is an evil little function
    # that installs packages via pip.
    # This means the script can install
    # it's own dependencies.
    try:
        __import__(package)
    except:
        import subprocess
        subprocess.call([sys.executable, "-m", "pip", "install", package])

install("colorama")
install("pathlib")
install("iterfzf")
install("prompt_toolkit")
install("munch")
install("appdirs")
install("art")
install("fire")

import os
import tkFileDialog
import getpass
import json
import shutil
import subprocess
import sys
import pprint
import art

# <snip> out all my business logic
print("Making Salad")

However I don't like this because it installs the packages to the global package repository. What I'd like is if all the packages were installed something like

scripts/python/make_salad/make_salad.py
                         /__packages__
                                      /colorama
                                      /pathlib
                                      /iterfzf
                                      ...
                                      /fire

and then this directory would be first on the search path when import is called. Is it possible to hack the above script so the above is possible?

Note the requirements

Only the single script is stored in repository
Users have to click on the script in windows explorer
Require to pip install external packages from within the script
Packages should not pollute the global packages

So you'd like to have things installed in a virtual environment of sorts? — 9769953, Aug 21 '19 at 13:48
Yeah but I can't have users messing around at the command line or they freak out. And I can't check any virtual environments into source control or prebuild anything. — bradgonesurfing, Aug 21 '19 at 13:52
Would any of the exe-builders help in this case (depending on the packages); in that it builds a single executable out of Python and all relevant packages. — 9769953, Aug 21 '19 at 13:55
maybe create the venv from the script? https://docs.python.org/3/library/venv.html — vinzenz, Aug 21 '19 at 13:58
Pff, that's a ridiculous bunch of requirements, if I may say so; and even with Python 2. Glad I'm not in your position. — 9769953, Aug 21 '19 at 13:58
@vinzBad The requirement (one of them) is Python 2.7; `venv` has only been built-in since Python 3.3, as per your link. — 9769953, Aug 21 '19 at 14:02
I can get virtualenv running from python2.7 via pip. It works quite well to script it and I now have a working solution. https://stackoverflow.com/a/57604352/158285 — bradgonesurfing, Aug 22 '19 at 07:46

bradgonesurfing · Accepted Answer · 2019-08-22T07:46:59.773

It turns out not too hard to use virtualenv directly from the script to manage this problem. I wrote a small helper class.

import sys
import subprocess

class App:
    def __init__(self, virtual_dir):
        self.virtual_dir = virtual_dir
        self.virtual_python = os.path.join(self.virtual_dir, "Scripts", "python.exe")

    def install_virtual_env(self):
        self.pip_install("virtualenv")
        if not os.path.exists(self.virtual_python):
            import subprocess
            subprocess.call([sys.executable, "-m", "virtualenv", self.virtual_dir])
        else:
            print("found virtual python: " + self.virtual_python)

    def is_venv(self):
        return sys.prefix==self.virtual_dir

    def restart_under_venv(self):
        print("Restarting under virtual environment " + self.virtual_dir)
        subprocess.call([self.virtual_python, __file__] + sys.argv[1:])
        exit(0)

    def pip_install(self, package):
        try:
            __import__(package)
        except:
            subprocess.call([sys.executable, "-m", "pip", "install", package, "--upgrade"])

    def run(self):
        if not self.is_venv():
            self.install_virtual_env()
            self.restart_under_venv()
        else:
            print("Running under virtual environment")

And can use it from the top of the main script make_salad.py like so

pathToScriptDir = os.path.dirname(os.path.realpath(__file__))

app = App(os.path.join(pathToScriptDir, "make_salad_virtual_env"))

app.run()

app.install("colorama")
app.install("pathlib")
app.install("iterfzf")
app.install("prompt_toolkit")
app.install("munch")
app.install("appdirs")
app.install("art")
app.install("fire")
app.install("appdirs")

print "making salad"

The first time running it will install the virtual env and all the required packages. Second time it will just run the script under the virtual environment.

found virtual python: C:\workspace\make_salad_virtualenv\Scripts\python.exe
Restarting under virtual environment C:\workspace\make_salad_virtualenv\Scripts
Running under virtual environment
Making Salad

score 2 · Answer 2 · answered Aug 21 '19 at 14:06

Perhaps you could use the --prefix option with pip, modify sys.path to include that prefix (+ lib/python2.7/site-packages/). I think --prefix will even work with a relative path:

def install(package):
    try:
        __import__(package)
    except:
        import subprocess
        subprocess.call([sys.executable, "-m", "pip", "install", package, "--prefix="packages"])
.
.
.
sys.path.append("packages/lib/python2.7/site-packages/")
.
.
.
import art
.
.
.

(untested)

How to call pip from a python script and make it install locally to that script?

2 Answers2