Activate Anaconda Python environment from makefile

Question

I want to use a makefile to build my project's environment using a makefile and anaconda/miniconda, so I should be able to clone the repo and simply run make myproject

myproject: build

build:
  @printf "\nBuilding Python Environment\n"
  @conda env create --quiet --force --file environment.yml
  @source /home/vagrant/miniconda/bin/activate myproject

If I try this, however, I get the following error

make: source: Command not found

make: *** [source] Error 127

I have searched for a solution, but [this question/answer(How to source a script in a Makefile?) suggests that I cannot use source from within a makefile.

This answer, however, proposes a solution (and received several upvotes) but this doesn't work for me either

( \
source /home/vagrant/miniconda/bin/activate myproject; \

)

/bin/sh: 2: source: not found

make: *** [source] Error 127

I also tried moving the source activate step to a separate bash script, and executing that script from the makefile. That doesn't work, and I assume for the a similar reason, i.e. I am running source from within a shell.

I should add that if I run source activate myproject from the terminal, it works correctly.

Read the contents of `/home/vagrant/miniconda/bin/activate`. Note that `source` is a shell built-in, not an executable to invoke, it executes the file's statements in the context of the current shell. Since `make` runs a different copy of shell, you cannot activate a virtualenv from it _in your current shell_, the same way you cannot change your current shell's directory from a makefile. — 9000, Aug 10 '16 at 15:55
So essentially there is no way for me to achieve this with a makefile? — Philip O'Brien, Aug 10 '16 at 15:56
You can clone a repo, activate an environment in a shell and run a make step (possibly another makefile) _in that invocation of shell_. E.g. `/bin/sh -c "source path/to/activate; make me_a_sandwich"` — 9000, Aug 10 '16 at 15:58
@PhilipO'Brien It would be really interesting to see what approach you picked in the end. I am facing the same limitations. — sorin, Sep 16 '16 at 13:21

score 16 · Answer 1 · answered May 29 '17 at 13:56

16

I had a similar problem; I wanted to create, or update, a conda environment from a Makefile to be sure my own scripts could use the python from that conda environment.
By default make uses sh to execute commands, and sh doesn't know source (also see this SO answer). I simply set the SHELL to bash and ended up with (relevant part only):

SHELL=/bin/bash
CONDAROOT = /my/path/to/miniconda2
.
.
install: sometarget
        source $(CONDAROOT)/bin/activate && conda env create -p conda -f environment.yml && source deactivate

Hope it helps

answered May 29 '17 at 13:56

Ludo

417
5
8

Big thanks for the tip - setting my shell to be the Bash shell was the trick to getting the `source` command to work! – ericmjl Aug 25 '17 at 14:08
Also using `&&` seems to be required to get the commands to run within environment. – conner.xyz Aug 29 '19 at 16:43
that really helps – AndreiR Sep 02 '20 at 14:24

score 3 · Answer 2 · edited Jul 06 '17 at 21:39

3

You should use this, it's functional for me at moment.

report.ipynb : merged.ipynb
    ( bash -c "source ${HOME}/anaconda3/bin/activate py27; which -a python; \
        jupyter nbconvert \
        --to notebook \
        --ExecutePreprocessor.kernel_name=python2 \
        --ExecutePreprocessor.timeout=3000 \
        --execute merged.ipynb \
        --output=$< $<" )

edited Jul 06 '17 at 21:39

Floern

33,559
24
104
119

answered Jul 06 '17 at 21:08

William Trigos

360
1
10

score 1 · Answer 3 · edited May 23 '17 at 11:46

I had the same problem. Essentially the only solution is stated by 9000. I have a setup shell script inside which I setup the conda environment (source activate python2), then I call the make command. I experimented with setting up the environment from inside Makefile and no success.

I have this line in my makefile:

installpy :
   ./setuppython2.sh && python setup.py install

The error messages is:

make
./setuppython2.sh && python setup.py install
running install
error: can't create or remove files in install directory

The following error occurred while trying to add or remove files in the
installation directory:

    [Errno 13] Permission denied: '/usr/lib/python2.7/site-packages/test-easy-install-29183.write-test'

Essentially, I was able to set up my conda environment to use my local conda that I have write access. But this is not picked up by the make process. I don't understand why the environment set up in my shell script using 'source' is not visible in the make process; the source command is supposed to change the current shell. I just want to share this so that other people don't wast time trying to do this. I know autotoools has a way of working with python. But the make program is probably limited in this respect.

My current solution is a shell script:

cat py2make.sh

#!/bin/sh

# the prefix should be change to the target
# of installation or pwd of the build system
PREFIX=/some/path
CONDA_HOME=$PREFIX/anaconda3
PATH=$CONDA_HOME/bin:$PATH
unset PYTHONPATH
export PREFIX CONDA_HOME PATH
source activate python2
make

This seems to work well for me.

There were a solution for similar situation but it does not seems to work for me:

My modified Makefile segment:

installpy :
   ( source activate python2; python setup.py install )

Error message after invoking make:

make
( source activate python2; python setup.py install )
/bin/sh: line 0: source: activate: file not found
make: *** [installpy] Error 1

Not sure where am I wrong. If anyone has a better solution please share it.

Activate Anaconda Python environment from makefile

3 Answers3

cat py2make.sh