Fully isolated Python environment

Question

Right now I am using virtualenv for my applications deployed in production.

I am running my apps like

cd $PROJECT_DIR
venv/bin/gunicorn -c gunicorn.conf.py my_app.wsgi:application

or

cd $PROJECT_DIR
venv/bin/celery worker --app=my_app.celery_tasks

Recently we have migrated to Python 3. The most recent Python 3.6 wasn't available for Ubuntu 14.04, so I compiled it by myself. Compiling it also allows me to benefit from optimizations using ./configure --enable-optimizations.

So I am thinking about always compiling Python by myself in my deployments. But at the same time keeping virtualenv file structure, so that the commands I am using for running apps in my virtual environment would remain the same.

I've seen people recommending using pyenv, but what I don't like about it is that it stores the Python itself in ~/.penv and, apparently, I need to fiddle with PATH environment variables to make my commands work in Cron and shell scripts, which I don't like. I'd like to keep all my environment in one directory if possible.

So my question is, can I somehow compile Python into venv directory in my project directory, so that the directory structure would be the same as when using virtualenv? Like:

$PROJECT_DIR/
    my_app/
    venv/
        bin/
            python
            python3.6
            celery
            gunicorn
            ...
        lib/
            python3.6/
                site-packages/
                    celery/
                    gunicorn/
                    ...

Afaik you can't do that. If you want such isolation, use Docker. — yedpodtrzitko, Jan 21 '17 at 15:00
I am quite sure it's possible. And not complicated. Just need some insights. — warvariuc, Jan 21 '17 at 15:41

Harald Nordgren · Accepted Answer · 2017-01-22T14:06:11.717

2

You can specify Python binary when creating the virutalenv:

virtualenv env -p /path/to/compiled/python3.6

edited Jan 22 '17 at 14:06

answered Jan 21 '17 at 17:10

Harald Nordgren

11,693
6
41
65

score 0 · Answer 2 · answered Jan 22 '17 at 15:00

Here is the script I came up with:

HERE=$( (cd -P $(dirname $0) && pwd) )
REQUIRED_PY_VERSION=$(cat ".python_version")
USR_DIR="$HERE/usr"
VENV_DIR="$HERE/venv"

echo "The system Python has another version $py_version"
echo "Downloading, compiling and installing the required version"
py_file_name="Python-$REQUIRED_PY_VERSION.tgz"
wget "https://www.python.org/ftp/python/$REQUIRED_PY_VERSION/$py_file_name" -O "$py_file_name"

py_source_dir="Python-$REQUIRED_PY_VERSION"
echo "Unpacking Python source code to $py_source_dir..."
mkdir -p "$py_source_dir"
tar xzvf "$py_file_name" --directory="$py_source_dir" --strip-components=1
cd "$py_source_dir"

# make clean
echo "Configuring compilation..."
./configure --enable-optimizations --prefix="$USR_DIR"
echo "Compiling..."
make
echo "Installing..."
make install

cd ..
echo "Removing downloaded files"
rm "$py_file_name"
rm -rf "$py_source_dir"
py_path="$USR_DIR/bin/python3"

echo "Creating virtual environment directory in $VENV_DIR"
"$py_path" -m venv "$VENV_DIR"

Fully isolated Python environment

2 Answers2