Activating a conda env inside a Docker container when using docker-compose to start Jupyter notebook

Question

I have the following Dockerfile.

FROM continuumio/miniconda3:4.5.11

# create a new user (defaults to 'al-khawarizmi')
USER root
ARG username=al-khawarizmi
RUN useradd --create-home --home-dir /home/${username} ${username}
ENV HOME /home/${username}

# switch to newly created user to avoid running container as root
USER ${username}
WORKDIR $HOME

# build and activate the specified conda environment from a file (defaults to 'environment.yml')
ARG environment=environment.yml
COPY ${environment} .
RUN conda env create --file ${environment} && \
    echo ". /opt/conda/etc/profile.d/conda.sh" >> ~/.bashrc && \ 
    echo "conda activate $(head -1 ${environment} | cut -d' ' -f2)" >> ~/.bashrc

The Dockerfile allows the user to specify a conda environment file as a build arg. Here would be a typical environment.yml file.

name: nessie-py

channels:
  - conda-forge
  - defaults

dependencies:
  - python=3.6
  - "notebook=5.7.*"
  - "matplotlib=3.0.*"
  - "numpy=1.15.*"
  - "pandas=0.23.*"

The user can run the image in the standard way and the conda environment will be automatically activated. Running

$ docker run -it image_name:image_tag

yields a bash prompt within the Docker container with the conda environment activated.

(environment_name)$

Now I would like to use docker-compose to start a Jupyter notebook server within the container (built with a conda environment file specifying Jupyter as a dependency).

When I use the following docker-compose.yml

version: "3.7"

services:
  notebook-server:
    build:
      context: ./
    ports:
      - "8888:8888"
    volumes:
      - ./:/home/al-khawarizmi
    command: jupyter notebook --no-browser ip=0.0.0.0

I get the following error.

$ docker-compose up
Creating network "nessie-py_default" with the default driver
Creating nessie-py_notebook-server_1 ... done
Attaching to nessie-py_notebook-server_1
notebook-server_1  | [FATAL tini (7)] exec jupyter failed: No such file or directory
nessie-py_notebook-server_1 exited with code 127

I suspected that this error meant that the conda environment is not activated. I then tried adding tty: true and stdin_open: true to the docker-compose.yml thinking that this should invoke and interactive bash prompt prior to running the command. This resulted in the same error as above.

I also tried defining a start-notebook.sh script that explicitly activates the conda environment prior to running the notebook.

#!/bin/bash
set -e

# activate the environment and start the notebook
conda activate nessie-py
jupyter notebook --no-browser ip=0.0.0.0

results in a different error

$ docker-compose up
Creating network "nessie-py_default" with the default driver
Creating nessie-py_notebook-server_1 ... done
Attaching to nessie-py_notebook-server_1
notebook-server_1  | 
notebook-server_1  | CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
notebook-server_1  | If your shell is Bash or a Bourne variant, enable conda for the current user with
notebook-server_1  | 
notebook-server_1  |     $ echo ". /opt/conda/etc/profile.d/conda.sh" >> ~/.bashrc
notebook-server_1  | 
notebook-server_1  | or, for all users, enable conda with
notebook-server_1  | 
notebook-server_1  |     $ sudo ln -s /opt/conda/etc/profile.d/conda.sh /etc/profile.d/conda.sh
notebook-server_1  | 
notebook-server_1  | The options above will permanently enable the 'conda' command, but they do NOT
notebook-server_1  | put conda's base (root) environment on PATH.  To do so, run
notebook-server_1  | 
notebook-server_1  |     $ conda activate
notebook-server_1  | 
notebook-server_1  | in your terminal, or to put the base environment on PATH permanently, run
notebook-server_1  | 
notebook-server_1  |     $ echo "conda activate" >> ~/.bashrc
notebook-server_1  | 
notebook-server_1  | Previous to conda 4.4, the recommended way to activate conda was to modify PATH in
notebook-server_1  | your ~/.bashrc file.  You should manually remove the line that looks like
notebook-server_1  | 
notebook-server_1  |     export PATH="/opt/conda/bin:$PATH"
notebook-server_1  | 
notebook-server_1  | ^^^ The above line should NO LONGER be in your ~/.bashrc file! ^^^
notebook-server_1  | 
notebook-server_1  | 
nessie-py_notebook-server_1 exited with code 1

This error suggests that bash is not sourcing ~/.bashrc prior to running the script.

I tried explicitly sourcing /opt/conda/etc/profile.d/conda.sh prior to activating the conda environment.

#!/bin/bash
set -e

# activate the environment and start the notebook
. /opt/conda/etc/profile.d/conda.sh
conda activate nessie-py
jupyter notebook --no-browser ip=0.0.0.0

which results in a different error!

$ docker-compose up
Creating network "nessie-py_default" with the default driver
Creating nessie-py_notebook-server_1 ... done
Attaching to nessie-py_notebook-server_1
notebook-server_1  | Could not find conda environment: nessie-py
notebook-server_1  | You can list all discoverable environments with `conda info --envs`.
notebook-server_1  | 
nessie-py_notebook-server_1 exited with code 1

I can check to see which conda envs are discoverable in the container by running

$ docker run -it nessie-py conda info --envs

which says that the environment does indeed exist.

$ docker run -it nessie-py_notebook-server conda info --envs
# conda environments:
#
nessie-py                /home/al-khawarizmi/.conda/envs/nessie-py
base                  *  /opt/conda

I am out of ideas at this point. This should be possible. Here is an example of a project with a docker-compose.yml file, a Dockerfile that specifies a conda environment and starts a Jupyter notebook server.

The additional complexities that I need include adding a non-root user to the Dockerfile and creating a new conda environment instead of updating the default base conda environment.

looks to me that the error comes from the `ip=0.0.0.0` argument, which should be `--ip=0.0.0.0` instead. — fernandezcuesta, Nov 13 '18 at 10:58
@fernandezcuesta thanks for catching the typo! But this doesn't not impact any of the error messages that I am receiving. The key problem seems to be in activating the conda environment. — davidrpugh, Nov 14 '18 at 09:02

fernandezcuesta · Accepted Answer · 2018-11-14T14:01:17.493

4

What happens is consequence of:

In the docker-compose.yml you've a typo in ip=0.0.0.0 which should be --ip=0.0.0.0 instead
Binding the host's folder into the container is overriding .bashrc. An easy change would be mounting into a subdirectory
You need to run bash in interactive mode (-i) so that .bashrc is properly read

As an example, changes on these points reflected in yourdocker-compose.yml:

version: "3.7"

    services:
      notebook-server:
        build:
          context: ./
        ports:
          - "8888:8888"
        volumes:
          - ./:/home/al-khawarizmi/hosthome
        command: bash -ic 'jupyter notebook --no-browser --ip=0.0.0.0'

edited Nov 14 '18 at 14:01

answered Nov 14 '18 at 13:18

fernandezcuesta

2,390
1
15
32

How did you figure out that binding the host's folder into the container is overriding `.bashrc`? – davidrpugh Nov 15 '18 at 06:01
it is described in [docker documentation](https://docs.docker.com/storage/bind-mounts/#mounting-into-a-non-empty-directory-on-the-container) – fernandezcuesta Nov 15 '18 at 10:07

score 1 · Answer 2 · answered May 05 '23 at 17:14

Dockerfile:

...

# Install conda
ENV PATH="/root/miniconda3/bin:${PATH}"

RUN wget \
    https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh \
    && mkdir /root/.conda \
    && bash Miniconda3-latest-Linux-x86_64.sh -b \
    && rm -f Miniconda3-latest-Linux-x86_64.sh

# Create the environment
COPY environment.yml .
RUN conda env update -f environment.yml --prune

# Override default shell and use bash
SHELL ["conda", "run", "/bin/bash", "-c"]

...

and then in docker-compose.yml:

command: conda run <your command>

e.g.:

command: conda run jupyter lab --ip 0.0.0.0 --port=8888 --no-browser --allow-root --NotebookApp.token='' --NotebookApp.password=''

score 0 · Answer 3 · answered Nov 12 '22 at 15:35

In accordance to this article, exists an easier way to do this, if you place SHELL ["conda", "run", "-n", "<venv>", "/bin/bash", "-c"] in your Dockerfile and use conda run --no-capture-output -n <venv> <your awesome command> in your docker-compose.yml.

For example your Dockerfile may looks like:

FROM continuumio/anaconda3

WORKDIR /usr/src/app

ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1

COPY ./environment.yml .
RUN conda env create -f environment.yml
SHELL ["conda", "run", "-n", "venv", "/bin/bash", "-c"]

COPY . .

And your docker-compose.yml may be similar to this:

version: '3.3'

services:

  web:
    environment: [.env]
    build: ./project
    ports:
      - 8000:8000
    command: conda run --no-capture-output -n venv uvicorn main:app --host 0.0.0.0 --reload
    volumes:
      - ./project:/usr/src/app

Activating a conda env inside a Docker container when using docker-compose to start Jupyter notebook

3 Answers3

Linked