Docker Image with Conda Env Activated and Ready for Shell Commands

Question

I have tried many ways through searching for a solution.

I think my problem is different.

I am wanting to have a docker image that has the environment installed and then active and ready for shell commands like: flake8, pylint, black, isort, coverage

Dockerfile

FROM continuumio/miniconda3

# Create the environment:
COPY conda_env_unit_tests.yml .
RUN conda env create -f conda_env_unit_tests.yml
RUN echo "conda activate up-and-down-pytorch" >> ~/.bashrc

conda_env_unit_test.yml

name: up-and-down-pytorch
channels:
  - defaults
  - conda-forge
  
dependencies:
  - python=3.9
  - pytest
  - pytest-cov
  - black
  - flake8
  - isort
  - pylint

.gitlab-ci.yml (slimmed down)

stages:
  - docker
  - linting
  - test

build_unit_test_docker:
  stage: docker
  tags:
    - docker
  image: docker:stable
  services:
    - docker:dind
  variables:
    IMAGE_NAME: "miniconda3-up-and-down-unit-tests"
  script:
    - cp /builds/upanddown1/mldl/up_and_down_pytorch/conda_env_unit_tests.yml /builds/upanddown1/mldl/up_and_down_pytorch/docker/unit_tests/
    - docker -D login $CI_REGISTRY -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD
    - docker -D build -t $CI_REGISTRY/upanddown1/mldl/up_and_down_pytorch/$IMAGE_NAME docker/unit_tests/
    - docker -D push $CI_REGISTRY/upanddown1/mldl/up_and_down_pytorch/$IMAGE_NAME
  rules:
    - changes:
      - docker/unit_tests/Dockerfile
      - conda_env_unit_tests.yml

unit-test:
  stage: test
  # image: continuumio/miniconda3:latest
  image: $CI_REGISTRY/upanddown1/mldl/up_and_down_pytorch/miniconda3-up-and-down-unit-tests
  script:
    # - conda env create --file conda_env.yml
    # - source activate up-and-down-pytorch
    - coverage run --source=. -m pytest --verbose
    - coverage report
    - coverage xml
  coverage: '/(?i)total.*? (100(?:\.0+)?\%|[1-9]?\d(?:\.\d+)?\%)$/'
  artifacts:
    reports:
      coverage_report:
        coverage_format: cobertura
        path: coverage.xml

The Docker Image gets uploaded to the gitlab registry and the unit test stage uses that image, however:

/bin/bash: line 127: coverage: command not found

(ultimate goal was to not have to create the conda environment every time I wanted to lint or run unit tests)

Can you use an ordinary Conda environment, or even a plain Python virtual environment? (Why is it important that you use Docker to run this shell in an isolated environment without access to host files?) — David Maze, Jan 29 '23 at 00:35
if I understand you correctly, it is because it will be faster if the environment is already setup instead of creating it from scratch every time? Sorry if I am misunderstanding you. — lr100, Jan 29 '23 at 19:19
One of Docker's core features is that containers can't normally see host files, and _vice versa_. Conversely, Python already has pretty good features for separating applications' library trees from each other. So I'm wondering if you can use an ordinary Python virtual environment for these sorts of developer tasks, and take Docker out of the picture entirely. — David Maze, Jan 30 '23 at 12:10
The flip side of this is that keeping a Conda or plain-Python virtual environment "activated" in Docker is a little tricky, since each `RUN` command launches a new shell and dotfiles like `.bashrc` aren't normally read. Some of the answers to [Activate conda environment in docker](https://stackoverflow.com/questions/55123637/activate-conda-environment-in-docker) describe the workarounds required. — David Maze, Jan 30 '23 at 12:12

lr100 · Accepted Answer · 2023-02-01T20:21:18.113

Figured it out today. Dropped the duration for the unit tests.

Change was to source the environment in the unit-test job. Didn't need to do that in the Dockerfile.

Dockerfile

FROM continuumio/miniconda3

# Create the environment:
COPY conda_env_unit_tests.yml .
RUN conda env create -f conda_env_unit_tests.yml

conda_env_unit_tests.yml

name: up-and-down-pytorch
channels:
  - defaults
  - conda-forge
  
dependencies:
  - python=3.9
  - pandas
  - pytest
  - pytest-cov
  - black
  - flake8
  - isort
  - pylint

.gitlab-ci.yml (slimmed down)

stages:
  - docker
  - linting
  - test

build_unit_test_docker:
  stage: docker
  tags:
    - docker
  image: docker:stable
  services:
    - docker:dind
  variables:
    IMAGE_NAME: "miniconda3-up-and-down-unit-tests"
  script:
    - cp /builds/upanddown1/mldl/up_and_down_pytorch/conda_env_unit_tests.yml /builds/upanddown1/mldl/up_and_down_pytorch/docker/unit_tests/
    - docker -D login $CI_REGISTRY -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD
    - docker -D build -t $CI_REGISTRY/upanddown1/mldl/up_and_down_pytorch/$IMAGE_NAME docker/unit_tests/
    - docker -D push $CI_REGISTRY/upanddown1/mldl/up_and_down_pytorch/$IMAGE_NAME
  rules:
    - changes:
      - docker/unit_tests/Dockerfile
      - conda_env_unit_tests.yml

unit-test:
  stage: test
  image: $CI_REGISTRY/upanddown1/mldl/up_and_down_pytorch/miniconda3-up-and-down-unit-tests
  script:
    - source activate up-and-down-pytorch
    - coverage run --source=. -m pytest --verbose
    - coverage report
    - coverage xml
  coverage: '/(?i)total.*? (100(?:\.0+)?\%|[1-9]?\d(?:\.\d+)?\%)$/'
  artifacts:
    reports:
      coverage_report:
        coverage_format: cobertura
        path: coverage.xml

Docker Image with Conda Env Activated and Ready for Shell Commands

1 Answers1