Install only available packages using "conda install --yes --file requirements.txt" without error

Question

While installing packages in requirements.txt using Conda through the following command

conda install --yes --file requirements.txt

If a package in requirements.txt is not available, then it throws a "No package error" such as the one shown below:

Using Anaconda Cloud api site https://api.anaconda.org

Fetching package metadata: ....

Error: No packages found in current linux-64 channels matching: nimfa ==1.2.3

You can search for this package on anaconda.org with
anaconda search -t conda nimfa ==1.2.3

Instead of throwing an error, is it possible to change this behavior such that it installs all the available packages in requirements.txt and throws a warning for those that are not available?

I would like this because, the package nimfa which the error says is not available, can be pip installed. So if I can change the behavior of conda install --yes --file requirements.txt to just throw a warning for unavailable packages, I can follow it up with the command pip install -r requirments.txt in .travis.yml so TravisCI attempts to install it from either place where it is available.

Till Hoffmann · Accepted Answer · 2018-07-21T17:59:56.210

223

I ended up just iterating over the lines of the file

$ while read requirement; do conda install --yes $requirement; done < requirements.txt

Edit: If you would like to install a package using pip if it is not available through conda, give this a go:

$ while read requirement; do conda install --yes $requirement || pip install $requirement; done < requirements.txt

Edit: If you are using Windows (credit goes to @Clay):

$ FOR /F "delims=~" %f in (requirements.txt) DO conda install --yes "%f" || pip install "%f"

edited Jul 21 '18 at 17:59

answered Jul 27 '16 at 09:56

Till Hoffmann

9,479
6
46
64

8

Thanks for this, small hint for anyone using it, make sure your requirements.txt ends with a newline, so that the final package is read in – Marawan Okasha Sep 22 '17 at 09:37
2

Is there a way to also automatically `pip install` packages which are not available using `conda install`, while iterating? – PeterB Sep 29 '17 at 22:43
2

@delusionX, I've extended the answer. – Till Hoffmann Sep 30 '17 at 15:46
I don't get it, why is that `conda install --yes --file requirements.txt` not work by itself? – Charlie Parker Oct 07 '17 at 17:31
1

@CharlieParker, I think the `--yes` flag only suppresses user interaction but does not ignore errors. – Till Hoffmann Oct 08 '17 at 12:44
15

@TillHoffmann 's solution on a Windows machine: `FOR /F "delims=~" %f in (requirements.txt) DO conda install --yes "%f" || pip install "%f"` Works very well. Thanks! – Clay Nov 11 '17 at 09:02
4

If you've comments in the requirements file, pip complains verbosely (annoying if harmless). A variation of the bash one-liner which ignores comments: `while read req; do if [[ $req != "#"* ]]; then conda install --yes $requirement || pip install $requirement; fi; done < requirements.txt` – drevicko Dec 30 '18 at 21:23
while is not recognised by conda :( what I am doing wrong here? – A.Papa Mar 02 '20 at 20:56
@A.Papa, the command should be executed in your local shell, e.g. bash. – Till Hoffmann Mar 03 '20 at 17:13
Once can add `-c conda-forge` to make sure conda-forge channel is searched as well. So it should be: `FOR /F "delims=~" %f in (requirements.txt) DO conda install -c conda-forge --yes "%f" || pip install "%f"` – e_kapti May 21 '20 at 20:09
1

This should be the default behavior. I can't believe this requires a "one liner bash script" when conda is the recommended way of installing python modules, only using pip as a fallback. – user3673 Aug 07 '21 at 15:30
For those (like me) who aren't familiar with bash scripts, you will only have to modify requirements.txt to the name of your requirements file. – jtb Sep 23 '21 at 15:01

score 18 · Answer 2 · edited Jul 17 '19 at 16:33

18

You can do this as mentioned in this

Export to .yml file

conda env export > freeze.yml

To reproduce:

conda env create -f freeze.yml

edited Jul 17 '19 at 16:33

jopasserat

5,721
4
31
50

answered Aug 16 '17 at 13:05

pbms

586
1
10
32

1

Here is archive.org link https://web.archive.org/web/20170704223211/https://www.continuum.io/content/conda-data-science – Talha Junaid Dec 18 '18 at 08:05
Looks like when using `conda env create -f file.yml` instead of `conda create -f file.yml` you cannot use the `--yes` flag to skip user prompted installation. Is there a `-y` equivalent for `conda env create`? – geominded Jan 05 '23 at 21:02

MarredCheese · Answer 3 · 2019-08-30T21:11:08.107

Pbms's answer here is the right way to do it, assuming you have an existing environment to copy off of. Conda is fully capable of installing both Conda packages and pip packages, as listed in environment.yml. I wanted to document the whole process in more detail. Note that I am using folder-based environments, which is why I added --prefix [path to environment folder] to most of the commands.

Say you installed an environment for an existing project to a folder called env in the current folder, like this:

conda create --prefix ./env

You'd generate environment.yml for that project's environment like this:

conda env export --prefix ./env > environment.yml

You'd create a new environment within some other folder by copying environment.yml to there and then running this from there:

conda env create --prefix ./env --file environment.yml

You'd get an already-existing environment to match environment.yml by once again copying environment.yml to there and then running this from there:

conda env update --prefix ./env --file environment.yml --prune

With the environment in question active, you'd verify the state of its packages like this:

conda list

This is an abridged version of what that command might print (note that the pip packages are marked pypi):

# Name                    Version                   Build  Channel
pip                       19.2.2                   py37_0
python                    3.7.4                h5263a28_0
numpy                     1.16.4           py37h19fb1c0_0
pandas                    0.25.1           py37ha925a31_0
pyodbc                    4.0.27           py37ha925a31_0
ibm-db                    3.0.1                    pypi_0    pypi
ibm-db-sa                 0.3.5                    pypi_0    pypi

Finally, this is an abridged version of what environment.yml might look like (note that the pip packages are listed in their own category):

dependencies:
  - pip=19.2.2=py37_0
  - python=3.7.4=h5263a28_0
  - numpy=1.16.4=py37h19fb1c0_0
  - pandas=0.25.1=py37ha925a31_0
  - pyodbc=4.0.27=py37ha925a31_0
  - pip:
    - ibm-db==3.0.1
    - ibm-db-sa==0.3.5

Be aware that using Conda and pip together can cause some heartburn because they can unknowingly blow away each other's dependencies. You are supposed to install all of your Conda packages first and then all of your pip packages afterward, rather than alternating between the two. If your environment breaks, the official recommendation is to delete and recreate it (from your environment.yml file). For more details, see this guide:

https://www.anaconda.com/using-pip-in-a-conda-environment/

score 1 · Answer 4 · answered Jul 18 '18 at 22:46

For those looking, I used this as @TillHoffmann 's solution for the fish shell:

$ while read requirement; conda install --yes $requirement; end < requirements.txt

And

$ while read requirement; conda install --yes $requirement;or pip install $requirement; end < requirements.txt

Install only available packages using "conda install --yes --file requirements.txt" without error

4 Answers4

Linked