Activate conda environment in docker

Question

I need to activate environment in docker and run a command in this environment. I create the environment, but then I try to activate this environment and run the command in this way:

CMD [ "source activate mro_env && ipython kernel install --user --name=mro_env" ]

but when I ran docker I get an error:

[FATAL tini (8)] exec source activate mro_env && ipython kernel install 
--user --name=mro_env failed: No such file or directory

This is the whole Dockerfile:

FROM continuumio/miniconda3

ADD /src/mro_env.yml /src/mro_env.yml
RUN conda env create -f /src/mro_env.yml

# Pull the environment name out of the mro_env.yml
RUN echo "source activate $(head -1 /src/mro_env.yml | cut -d' ' -f2)" > ~/.bashrc
ENV PATH /opt/conda/envs/$(head -1 /src/mro_env.yml | cut -d' ' -f2)/bin:$PATH

CMD [ "source activate mro_env && ipython kernel install --user --name=mro_env" ]

Answer 1

Followed this tutorial and it worked. Example Dockerfile:

FROM continuumio/miniconda
WORKDIR /usr/src/app
COPY ./ ./
RUN conda env create -f environment.yml

# Make RUN commands use the new environment:
SHELL ["conda", "run", "-n", "myenv", "/bin/bash", "-c"]

EXPOSE 5003
# The code to run when container is started:
ENTRYPOINT ["conda", "run", "-n", "myenv", "python3", "src/server.py"]

Update:

You can use "conda run --no-capture-output" to not buffer IO if you use the 4.9 version of conda. Updated Dockerfile:

FROM continuumio/miniconda
WORKDIR /usr/src/app
COPY ./ ./
RUN conda env create -f environment.yml

# Make RUN commands use the new environment:
SHELL ["conda", "run", "--no-capture-output", "-n", "myenv", "/bin/bash", "-c"]

EXPOSE 5003
# The code to run when container is started:
ENTRYPOINT ["conda", "run", "--no-capture-output", "-n", "myenv", "python3", "src/server.py"]

Answer 2

For me, the solution introduced here worked seemlessly:

FROM continuumio/miniconda3
RUN conda create -n {env} python=3.6
RUN echo "source activate {env}" > ~/.bashrc
ENV PATH /opt/conda/envs/{env}/bin:$PATH

UPD: 'env' is just an example, adjust 'env' name if needed/required.

Answer 3

You can set CONDA_DEFAULT_ENV

Like this:

FROM continuumio/miniconda3

ARG conda_env=mro_env

ADD /src/environment.yml /src/environment.yml
RUN conda env create -f /src/environment.yml

ENV PATH /opt/conda/envs/$conda_env/bin:$PATH
ENV CONDA_DEFAULT_ENV $conda_env

CMD [ "python", "test.py" ]

UPDATE:

Better use activate. Work for me:

FROM continuumio/miniconda3

ADD /src/environment.yml /src/environment.yml

RUN conda env create -f /src/environment.yml
ENV PATH /opt/conda/envs/mro_env/bin:$PATH
RUN /bin/bash -c "source activate mro_env"

CMD [ "python", "test.py" ]

Answer 4

The problem for me was that running the command conda activate env inside docker after installing caused conda to ask me to use the conda init bash command. However, this command asks you to restart the shell, which we don't want to do inside docker. So the solution is to realize that the reason conda is asking you to restart the shell is because it has modified and wants to reload the contents of ~/.bashrc. We can do this manually and forego the need for restarting the shell using:

. ~/.bashrc

Here's the full dockerfile for those who want it:

FROM ubuntu:18.04
# update apt and get miniconda
RUN apt-get update \
    && apt-get install -y wget \
    && wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh


# install miniconda
ENV PATH="/root/miniconda3/bin:$PATH"
RUN mkdir /root/.conda && bash Miniconda3-latest-Linux-x86_64.sh -b

# create conda environment
RUN conda init bash \
    && . ~/.bashrc \
    && conda create --name test-env python=3.7 \
    && conda activate test-env \
    && pip install ipython

Edit - deal with comment

The commenter is correct in that the above example doesn't work for switching conda environments within docker. More recently, I figured out that to get this to work, you should start every RUN command with conda init bash && .~/.bashrc && conda activate env.

Here's another example:

FROM quay.io/pypa/manylinux2014_x86_64

RUN yum install -y wget
RUN wget https://repo.anaconda.com/miniconda/Miniconda3-py39_4.9.2-Linux-x86_64.sh
RUN bash Miniconda3-py39_4.9.2-Linux-x86_64.sh -b -p /Miniconda3

RUN /Miniconda3/bin/conda create -y --name py37 python=3.7 pytest
RUN /Miniconda3/bin/conda create -y --name py38 python=3.8 pytest
RUN /Miniconda3/bin/conda create -y --name py39 python=3.9 pytest

RUN /Miniconda3/bin/conda init && bash ~/.bashrc && . ~/.bashrc

ENV conda /Miniconda3/bin/conda
ENV bashrc /root/.bashrc

# install numpy in each env
RUN $conda init && . $bashrc && conda activate py37 && pip install numpy 
RUN $conda init && . $bashrc && conda activate py38 && pip install numpy
RUN $conda init && . $bashrc && conda activate py39 && pip install numpy

Answer 5

As the user merv points out in one of the comments above (sorry don't have enough rep to vote up the comment) conda run buffers all stdout/stderr, thus making it not workable for applications interacting or even just displaying logs over I/O.

I noticed there was no accepted answer, so I just post what has worked very well for me:

You can use an entrypoint script to activate the conda enviroment and let it take over further input from the Dockerfile such that the python script can be executed within the activated conda environment

Example Dockerfile:

FROM continuumio/miniconda3

# make docker use bash instead of sh
SHELL ["/bin/bash", "--login", "-c"]

# copy all necessary files
COPY environment.yml .
COPY ownchain/mypyscript.py .
COPY entrypoint.sh /usr/local/bin/

# make entrypoint script executable
RUN chmod u+x /usr/local/bin/entrypoint.sh
# create environment
RUN conda env create -f environment.yml

ENTRYPOINT ["/usr/local/bin/entrypoint.sh"]
CMD ["python", "mypyscript.py"]

Where the entrypoint.sh looks like this:

#!/bin/bash --login
set -e

# activate conda environment and let the following process take over
conda activate myenv
exec "$@"

All credit to David R. Pugh from this post that has more details, in particular with regards to Jupyter.

Answer 6

Here is how to run a script in a conda environment without buffering all standard input/output.

The --no-capture-output option is available since conda version 4.9.

FROM continuumio/miniconda3

COPY ./environment.yml ./environment.yml

RUN conda env create && conda clean --all -f --yes

ENTRYPOINT ["conda", "run", "--no-capture-output", "-n", "the_environment_name", "python", "myscript.py"]

Answer 7

I understand that there is no one solution fits all but this is what I have been using with my Flask applications:

FROM continuumio/miniconda3

COPY environment.yml .
RUN conda env create -f environment.yml

COPY app /app
WORKDIR /app

CMD ["conda", "run", "-n", "my_env", "python", "app.py"]

The logic is very simple. First, environment file is copied followed by the creation of the environment. Next, application files (this is where all my Flask application files are located) are copied. Finally, using CMD the application is started by pointing to the environment.

This is the project directory structure I used with the Dockerfile:

-- project
    -- app
        -- app.py
    -- environment.yaml
    -- Dockerfile

Note that this does not activate the environment per se but points to it at run time in the CMD command

Answer 8

I stumbled upon this question while trying to activate an environment and then installing some packages inside it. The SHELL solution did not work for me (maybe because I was attempting this on an older version of conda - 4.5.4 to be precise).

The solution is to activate the environment and run all required commands inside a new shell. Also, remember that each RUN will start a new shell that will not remember anything from the previous RUN.

FROM continuumio/miniconda3

ADD /src/environment.yml /src/environment.yml

RUN conda env create -f /src/environment.yml
ENV PATH /opt/conda/envs/mro_env/bin:$PATH
RUN /bin/bash -c "source activate mro_env \
    && conda config --add channels conda-forge \
    && conda install Jupiter \
    && conda env list"

CMD [ "python", "test.py" ]

Note every command is within "" of the same /bin/bash -c.

Answer 9

Similar to other answers but with SHELL which looks more cleaner

RUN wget \
https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh \
&& mkdir /root/.conda \
&& bash Miniconda3-latest-Linux-x86_64.sh -b \
&& rm -f Miniconda3-latest-Linux-x86_64.sh \
&& echo "source activate base" > ~/.bashrc

SHELL ["/bin/bash", "-c"]
ARG CONDA_ENV=env_name 
# Create env
RUN conda --version \
  && conda create -n ${CONDA_ENV} python=3.7 \
  && source activate ${CONDA_ENV}

Answer 10

It is pretty hopeless to get this working reliably. Sometimes a solution works on a system for a while and then suddenly stops working. There is also little to now support for this from the continuum/minicoda3 creators. I guess, because they intend to have the container used differently. I tried the following and it runs through, however, when trying to run the container with docker-compose it looks for the environment at a weird location /opt/conda/envs/ instead of /root/miniconda3/envs/.

FROM ubuntu:22.04
ENV PATH="/root/miniconda3/bin:${PATH}"
ARG PATH="/root/miniconda3/bin:${PATH}"
RUN apt-get update

RUN apt-get install -y wget build-essential time && rm -rf /var/lib/apt/lists/*

RUN wget \
    https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh \
    && mkdir /root/.conda \
    && bash Miniconda3-latest-Linux-x86_64.sh -b \
    && rm -f Miniconda3-latest-Linux-x86_64.sh 

RUN conda --version

RUN conda init bash

RUN conda create -n myenv -c conda-forge -c bioconda python==3.10 pip

SHELL ["conda", "run", "-n", "myenv", "/bin/bash", "-c"]

COPY requirements.txt requirements.txt

RUN pip install -r requirements.txt

Answer 11

I use this to activate an environment when I run a container. In order to install packages into that environment I just use "-n myenv".

ENV PATH /opt/miniconda/bin:${PATH}

RUN wget --quiet https://repo.anaconda.com/miniconda/Miniconda3-4.5.11-Linux-x86_64.sh -O ~/miniconda.sh --no-check-certificate && /bin/bash ~/miniconda.sh -b -p /opt/miniconda
RUN rm ~/miniconda.sh
RUN conda clean -ya

RUN conda update -n base -c defaults conda
RUN conda create --name detic python=3.8 -y

RUN echo ". /opt/miniconda/etc/profile.d/conda.sh" >> ~/.bashrc
RUN echo "conda activate detic" >> ~/.bashrc

RUN conda install -n detic ipython

I don't like to change the entrypoints

Answer 12

You can use this solution with docker-compose.

In accordance to this article, exists an easier way to do this, if you place SHELL ["conda", "run", "-n", "<venv>", "/bin/bash", "-c"] in your Dockerfile and use conda run --no-capture-output -n <venv> <your awesome command> in your docker-compose.yml.

Answer 13

If you don't need to change environments away from the base you could also do this:

COPY conda.yaml /
RUN { echo "name: base"; tail +2 /conda.yaml; } > /base.yaml
RUN conda env update --file /base.yaml --prune

The environment in conda.yaml could have any name since we replace it with base.

Answer 14

Since, conda run is an experimental feature the correct way is to add this line to your Dockerfile

SHELL [ "/bin/bash", "--login", "-c" ]

after this you can continue with

RUN conda init bash

and then continue to activate your environment with

RUN conda activate <env_name>