How to use virtualenv in makefile

Question

I want to perform several operations while working on a specified virtualenv.

For example command

make install

would be equivalent to

source path/to/virtualenv/bin/activate
pip install -r requirements.txt

Is it possible?

Answer 1

I like using something that runs only when requirements.txt changes:

This assumes that source files are under project in your project's root directory and that tests are under project/test. (You should change project to match your actually project name.)

venv: venv/touchfile

venv/touchfile: requirements.txt
    test -d venv || virtualenv venv
    . venv/bin/activate; pip install -Ur requirements.txt
    touch venv/touchfile

test: venv
    . venv/bin/activate; nosetests project/test

clean:
    rm -rf venv
    find -iname "*.pyc" -delete

• Run make to install packages in requirements.txt.
• Run make test to run your tests (you can update this command if your tests are somewhere else).
• run make clean to delete all artifacts.

Answer 2

In make you can run a shell as command. In this shell you can do everything you can do in a shell you started from comandline. Example:

install:
    ( \
       source path/to/virtualenv/bin/activate; \
       pip install -r requirements.txt; \
    )

Attention must be paid to the ;and the \.

Everything between the open and close brace will be done in a single instance of a shell.

Answer 3

Normally make runs every command in a recipe in a different subshell. However, setting .ONESHELL: will run all the commands in a recipe in the same subshell, allowing you to activate a virtualenv and then run commands inside it.

Note that .ONESHELL: applies to the whole Makefile, not just a single recipe. It may change behaviour of existing commands, details of possible errors in the full documentation. This will not let you activate a virtualenv for use outside the Makefile, since the commands are still run inside a subshell.

Reference documentation: https://www.gnu.org/software/make/manual/html_node/One-Shell.html

Example:

.ONESHELL:

.PHONY: install
install:
    source path/to/virtualenv/bin/activate
    pip install -r requirements.txt

Answer 4

I have had luck with this.

install:
    source ./path/to/bin/activate; \
    pip install -r requirements.txt; \

Answer 5

This is an alternate way to run things that you want to run in virtualenv.

BIN=venv/bin/

install:
    $(BIN)pip install -r requirements.txt

run:
    $(BIN)python main.py

PS: This doesn't activate the virtualenv, but gets thing done. Hope you find it clean and useful.

Answer 6

I like to set my Makefile up so that it uses a .venv directory if one exists, but defaults to using the PATH.

For local development, I like to use a virtual environment, so I run:

# Running this:  # Actually runs this:
make venv        # /usr/bin/python3 -m venv .venv
make deps        # .venv/bin/python setup.py develop
make test        # .venv/bin/python -m tox

If I'm installing into a container though, or into my machine, I might bypass the virtual environment by skipping make venv:

# Running this:  # Actually runs this:
make deps        # /usr/bin/python3 setup.py develop
make test        # /usr/bin/python3 -m tox

Setup

At the top of your Makefile, define these variables:

VENV           = .venv
VENV_PYTHON    = $(VENV_PYTHON)/bin/python
SYSTEM_PYTHON  = $(or $(shell which python3), $(shell which python))
# If virtualenv exists, use it. If not, find python using PATH
PYTHON         = $(or $(wildcard $(VENV_PYTHON)), $(SYSTEM_PYTHON))

If ./.venv exists, you get:

VENV          = .venv
VENV_PYTHON   = .venv/bin/python
SYSTEM_PYTHON = /usr/bin/python3
PYTHON        = .venv/bin/python

If not, you get:

VENV          = .venv
VENV_PYTHON   = .venv/bin/python
SYSTEM_PYTHON = /usr/bin/python3
PYTHON        = /usr/bin/python3

Note: /usr/bin/python3 could be something else on your system, depending on your PATH.

Then, where needed, run python stuff like this:

$(PYTHON) -m tox
$(PYTHON) -m pip ...

You might want to create a target called "venv" that creates the .venv directory:

$(VENV_PYTHON):
    rm -rf $(VENV)
    $(SYSTEM_PYTHON) -m venv $(VENV)

venv: $(VENV_PYTHON)

And a deps target to install dependencies:

deps:
    $(PYTHON) setup.py develop
    # or whatever you need:
    #$(PYTHON) -m pip install -r requirements.txt

Example

Here's my Makefile:

# Variables
VENV           = .venv
VENV_PYTHON    = $(VENV)/bin/python
SYSTEM_PYTHON  = $(or $(shell which python3), $(shell which python))
PYTHON         = $(or $(wildcard $(VENV_PYTHON)), $(SYSTEM_PYTHON))

## Dev/build environment

$(VENV_PYTHON):
    rm -rf $(VENV)
    $(SYSTEM_PYTHON) -m venv $(VENV)

venv: $(VENV_PYTHON)

deps:
    $(PYTHON) -m pip install --upgrade pip
    # Dev dependencies
    $(PYTHON) -m pip install tox pytest
    # Dependencies
    $(PYTHON) setup.py develop

.PHONY: venv deps

## Test

test:
    $(PYTHON) -m tox

.PHONY: test

## Build source distribution, install

sdist:
    $(PYTHON) setup.py sdist

install:
    $(SYSTEM_PYTHON) -m pip install .

.PHONY: build install

Answer 7

Based on the answers above (thanks @Saurabh and @oneself!) I've written a reusable Makefile that takes care of creating virtual environment and keeping it updated: https://github.com/sio/Makefile.venv

It works by referencing correct executables within virtualenv and does not rely on the "activate" shell script. Here is an example:

test: venv
    $(VENV)/python -m unittest

include Makefile.venv

Differences between Windows and other operating systems are taken into account, Makefile.venv should work fine on any OS that provides Python and make.

Answer 8

A bit late to the party but here's my usual setup:

# system python interpreter. used only to create virtual environment
PY = python3
VENV = venv
BIN=$(VENV)/bin

# make it work on windows too
ifeq ($(OS), Windows_NT)
    BIN=$(VENV)/Scripts
    PY=python
endif


all: lint test

$(VENV): requirements.txt requirements-dev.txt setup.py
    $(PY) -m venv $(VENV)
    $(BIN)/pip install --upgrade -r requirements.txt
    $(BIN)/pip install --upgrade -r requirements-dev.txt
    $(BIN)/pip install -e .
    touch $(VENV)

.PHONY: test
test: $(VENV)
    $(BIN)/pytest

.PHONY: lint
lint: $(VENV)
    $(BIN)/flake8

.PHONY: release
release: $(VENV)
    $(BIN)/python setup.py sdist bdist_wheel upload

clean:
    rm -rf $(VENV)
    find . -type f -name *.pyc -delete
    find . -type d -name __pycache__ -delete

I did some more detailed writeup on that, but basically the idea is that you use the system's Python to create the virtual environment and for the other targets just prefix your command with the $(BIN) variable which points to the bin or Scripts directory inside your venv. This is equivalent to the activate function.

Answer 9

You also could use the environment variable called "VIRTUALENVWRAPPER_SCRIPT". Like this:

install:
    ( \
       source $$VIRTUALENVWRAPPER_SCRIPT; \
       pip install -r requirements.txt; \
    )

Answer 10

I found prepending to $PATH and adding $VIRTUAL_ENV was the best route:

• No need to clutter up recipes with activate and constrain oneself to ; chaining
  • Shown here and here
• Can simply use python as you would normally, and it will fall back onto system Python
  • No need for third party packages
  • Compatible with both Windows (if using bash) and POSIX

# SYSTEM_PYTHON defaults to Python on the local machine
SYSTEM_PYTHON = $(shell which python)

REPO_ROOT = $(shell pwd)
# Specify with REPO_ROOT so recipes can safely change directories
export VIRTUAL_ENV := ${REPO_ROOT}/venv
# bin = POSIX, Scripts = Windows
export PATH := ${VIRTUAL_ENV}/bin:${VIRTUAL_ENV}/Scripts:${PATH}

And for those interested in example usages:

# SEE: http://redsymbol.net/articles/unofficial-bash-strict-mode/
SHELL=/bin/bash -euo pipefail
.DEFAULT_GOAL := fresh-install

show-python:    ## Show path to python and version.
    @echo -n "python location: "
    @python -c "import sys; print(sys.executable, end='')"
    @echo -n ", version: "
    @python -c "import platform; print(platform.python_version())"

show-venv: show-python
show-venv:  ## Show output of python -m pip list.
    python -m pip list

install: show-python
install:    ## Install all dev dependencies into a local virtual environment.
    python -m pip install -r requirements-dev.txt --progress-bar off

fresh-install:  ## Run a fresh install into a local virtual environment.
    -rm -rf venv
    $(SYSTEM_PYTHON) -m venv venv
    @$(MAKE) install

Answer 11

You should use this, it's functional for me at moment.

report.ipynb : merged.ipynb
    ( bash -c "source ${HOME}/anaconda3/bin/activate py27; which -a python; \
        jupyter nbconvert \
        --to notebook \
        --ExecutePreprocessor.kernel_name=python2 \
        --ExecutePreprocessor.timeout=3000 \
        --execute merged.ipynb \
        --output=$< $<" )