Containerize a conda environment in a Singularity container

Question

I've come across several instances where it would be really helpful to containerize a conda environment for long-term reproducibility. As I'm normally running in high-performance computing systems, they need to be Singularity containers for security reasons. How can this be done?

Answer 1

First, you'll want to get the environment YML for your particular conda environment.

conda activate your_env
conda env export > environment.yml

Here's an example Singularity recipe (in file named 'Singularity' in same directory as 'environment.yml'):

Bootstrap: docker

From: continuumio/miniconda3

%files
    environment.yml

%post
    /opt/conda/bin/conda env create -f environment.yml

%runscript
    exec /opt/conda/envs/$(head -n 1 environment.yml | cut -f 2 -d ' ')/bin/"$@"

Build this with

sudo singularity build conda.sif Singularity

Now, you'll have a functioning container using libraries from your conda environment that can be run anywhere you have Singularity installed!

Example:

singularity run conda.sif ipython

Notes:

Depending on the version of Singularity you're using, you may need to alter $(head -n 1 environment.yml | cut -f 2 -d ' ') with the name of your environment.

Since you can't activate the environment from the runscript, you'll be restricted to binaries installed in your particular environment with the provided runscript.

Answer 2

I have found it useful because you can have a container with anaconda3 environment installed and easily create new Environments for your different projects whenever you want.

This is simple and I will go over it step by step:

1. Create the container in your local machine by the following definition file (you can name it whatever you want. Note that some lines might be avoidable ):

   Bootstrap: library
   From: ubuntu:18.04
   Stage: build
   
   %post
   
   apt-get update && apt-get -y upgrade
   apt-get -y install \
   build-essential \
   wget \
   bzip2 \
   ca-certificates \
   libglib2.0-0 \
   libxext6 \
   libsm6 \
   libxrender1 \
   git
   rm -rf /var/lib/apt/lists/*
   apt-get clean
   #Installing Anaconda 3 
   wget -c https://repo.anaconda.com/archive/Anaconda3-2020.02-Linux-x86_64.sh
   /bin/bash Anaconda3-2020.02-Linux-x86_64.sh -bfp /usr/local
   #Conda configuration of channels from .condarc file
   conda config --file /.condarc --add channels defaults
   conda config --file /.condarc --add channels conda-forge
   conda update conda
   #List installed environments
   conda list
   

2. Then, in order to build the container, run the following commands:

   sudo singularity build ContainerName.sif YourDefineFile.def
   

3. Now you can create your conda env(you can use common ways or create it by YML file which is an exported file from an existing environment)

   For example, I did it with a YML file: First, you need to get into your .sif container as following:

    Singularity shell YourContainerName.sif 
   

   And then:

    conda env create --name envname --file=YourEnvironments.yml
   

4. Therefore, After your env is created, you can activate it with the following commands(again first you need to jump into your container):

   singularity shell YourContainer.Sif
   
   source activate YourEnvName
   

Answer 3

Using conda-pack it is possible to containerize existing conda environments without re-creating them from environment.yml. This is particularly useful when the environment doesn't resolve anymore, or when packages have been installed into the environment without conda, e.g. using R's install.packages.

1. pack the environment

   conda-pack -n <MY_ENV> -o packed_environment.tar.gz
   

2. create this Singularity file

   
   Bootstrap: docker
   
   From: continuumio/miniconda3
   
   %files
       packed_environment.tar.gz /packed_environment.tar.gz
   
   %post
       tar xvzf /packed_environment.tar.gz -C /opt/conda
       conda-unpack
       rm /packed_environment.tar.gz
   

3. build the image

   singularity build --fakeroot <OUTPUT_CONTAINER.sif> Singularity
   

For more details, including a Docker/Podman variant, check out my grst/containerize-conda repository.

Limitations: The approach probably only works if the source environment is on a linux x64 machine.