208

I usually use conda to manage my environments, but now I am on a project that needs a little more horsepower than my laptop. So I am trying to use my university's workstations which have new Intel Xeons. But I don't have admin rights and the workstation does not have conda so I am forced to work with virtualenv and pip3.

How do I generate a requirements.txt from conda that will work with pip3 and venv?

conda list -e > requirements.txt

does not generate a compatible file:

= is not a valid operator. Did you mean == ?

The conda output is:

# This file may be used to create an environment using:
# $ conda create --name <env> --file <this file>
# platform: osx-64
certifi=2016.2.28=py36_0
cycler=0.10.0=py36_0
freetype=2.5.5=2
icu=54.1=0
libpng=1.6.30=1
matplotlib=2.0.2=np113py36_0
mkl=2017.0.3=0
numpy=1.13.1=py36_0
openssl=1.0.2l=0
pip=9.0.1=py36_1
pyparsing=2.2.0=py36_0
pyqt=5.6.0=py36_2
python=3.6.2=0
python-dateutil=2.6.1=py36_0
pytz=2017.2=py36_0
qt=5.6.2=2
readline=6.2=2
scikit-learn=0.19.0=np113py36_0
scipy=0.19.1=np113py36_0
setuptools=36.4.0=py36_1
sip=4.18=py36_0
six=1.10.0=py36_0
sqlite=3.13.0=0
tk=8.5.18=0
wheel=0.29.0=py36_0
xz=5.2.3=0
zlib=1.2.11=0

I thought I would just manually change all = to == but the there are two = in the conda output. Which one to change? Surely there is an easier way?

EDIT: pip freeze > requirements.txt gives:

certifi==2016.2.28
cycler==0.10.0
matplotlib==2.0.2
matplotlib-venn==0.11.5
numpy==1.13.1
pyparsing==2.2.0
python-dateutil==2.6.1
pytz==2017.2
scikit-learn==0.19.0
scipy==0.19.1
six==1.10.0
vvvvv
  • 25,404
  • 19
  • 49
  • 81
ITA
  • 3,200
  • 3
  • 18
  • 32
  • 9
    you can use 'pip freeze' on your computer – Evgeny Jun 09 '18 at 19:41
  • 1
    But the outputs are different, I mean the conda command has more packages than output of `pip freeze`. – ITA Jun 09 '18 at 19:45
  • that looks a bit strange, but usually you can edit the rewuiremrnts.txt by hand as last resort. another option in pipenv, a wrapper around pip and virtualenv – Evgeny Jun 09 '18 at 19:53
  • 2
    Nope, none if it is working. I find it so strange that this can't be done. Yeah maybe its uncommon for people to switch between environment managers, but still ... the need _is_ conceivable. – ITA Jun 09 '18 at 20:10
  • care to [pick an answer](https://stackoverflow.com/help/someone-answers)? – Evgeny Jun 15 '18 at 22:39
  • 3
    I'm not sure, I don't see a full answer; in the sense you have explained what the tools do, but there is no solution as far as I see in _how_ to replicate a conda environment successfully with virtualenv of pipenv. – ITA Jun 17 '18 at 15:11
  • totally up to you ! – Evgeny Jun 17 '18 at 17:37
  • basically the workflow is that you recreate `conda` list of packages with `conda` and `pip` generated files with with `pip`, rarely you have a setup where you do have `conda` installed and no `pip` available to generate `requirements.txt` – Evgeny Jun 17 '18 at 19:45
  • I don't think you understand...with anconda you get access to pip as well. So yes, what you say is right...access to anconda but no pip is next to zero chance. But you **can** have a situation where you have access to conda and pip on _one_ machine but no conda just `virtualenv, pip, venv, pipenv, etc` on _another_ machine. Question is how to port an environment from one machine to another. None of the suggested methods actually worked. – ITA Jun 17 '18 at 20:56
  • If both machines have `pip`, you can port the environement easily. Perhaps you attach a different meaning to environment - it is a set of packages, not the python installation. Is that what worries you, the python version? – Evgeny Jun 17 '18 at 21:04
  • No. (1) The versioning from anaconda to pip in incompatible, (2) the process is not automatic. Of course I can install each package listed by anaconda by hand in pip. But that is a terrible way to do it. – ITA Jun 18 '18 at 16:20
  • There is a machine that has no `pip`? it is a standard installation. You port things with pip, leaving conda alone. – Evgeny Jun 18 '18 at 18:53
  • Like i said _versioning_ is incompatible. And process is not _automatic_. I never talked about a machine with no `pip`. Also I did say yes you can do it manually package by package, but that is a _terrible_ way to do it. – ITA Jun 18 '18 at 20:36
  • conda does not perform the same role as pip; conda manages a 'bigger picture', pip specifically manages Python dependencies. Your conda list includes such packages as `libpng`, `icu` and `openssl` for example, which are well outside the remit of pip. See [What is the difference between pip and conda?](//stackoverflow.com/a/20994790). Basically anything without a `=py36_*` or `=np*py36_*` dependency specifier is a conda requirement that lives *next* to Python, not in the Python library ecosystem itself. – Martijn Pieters Jul 19 '18 at 10:15
  • Stack Overflow is not a tool to make feature requests to a third-party developer, nor would your feature request be feasible. Translating the conda requirements list to a different environment without conda is a huge task, and not really attainable when you think about it a bit longer; you'd basically be re-implementing conda for each target environment! You didn't tell us if those workstations run Windows or Linux or some other OS, for example. You didn't tell us if you need a GUI environment to render matplotlib graphs and interact with Jupyter notebooks. That matters here! – Martijn Pieters Jul 19 '18 at 10:25
  • @MartijnPieters So you are saying Python development is so fragmented that its basically "pick your poison and live with it" no matter come "hell or high water"? Yes i understand that there are nitty bitty technicalities, its not something that _cannot_ be overcome. – ITA Jul 19 '18 at 16:28

7 Answers7

351

As the comment at the top indicates, the output of

conda list -e > requirements.txt

can be used to create a conda virtual environment with

conda create --name <env> --file requirements.txt

but this output isn't in the right format for pip.

If you want a file which you can use to create a pip virtual environment (i.e. a requirements.txt in the right format) you can install pip within the conda environment, then use pip to create requirements.txt.

conda activate <env>
conda install pip
pip freeze > requirements.txt

Then use the resulting requirements.txt to create a pip virtual environment:

python3 -m venv env
source env/bin/activate
pip install -r requirements.txt

When I tested this, the packages weren't identical across the outputs (pip included fewer packages) but it was sufficient to set up a functional environment.

For those getting odd path references in requirements.txt, use:

pip list --format=freeze > requirements.txt
rmwenz
  • 3,665
  • 1
  • 8
  • 8
  • 19
    This is the best answer and should be the accepted answer, because it provides specific commands to use to accomplish / answer the original SO question. – Rich Lysakowski PhD Mar 23 '20 at 03:30
  • This works. You can also use `conda create -n ` `conda activate ` `pip install -r requirements.txt` – JohnnyUtah May 19 '20 at 21:58
  • 35
    For those following this thread and getting odd path references in `requirements.txt`, use `pip list --format=freeze > requirements.txt` instead. Ref: https://stackoverflow.com/a/62886215/13095028 – tnwei Feb 05 '21 at 02:17
  • 2
    @JohnnyUtah careful there. Creating a totally empty environment won't include `pip` and so your recommendation would end up installing all those packages *outside* the intended target environment. Besides, the OP clearly is asking about moving to a system that doesn't have Conda in the first place, hence the turn to `pip` and `venv`. – merv Apr 12 '21 at 03:36
  • 2
    Ah good point @merv. Could you explicitly install `pip` after creating and activating the Conda environment and before running `pip install -r requirements.txt`? – JohnnyUtah Apr 14 '21 at 15:48
  • 2
    @JohnnyUtah sure, or just specify it with the create: `condo create -n foo python=3.8 pip` or [use a YAML like this](https://stackoverflow.com/a/59056234/570918) (without the `anaconda`). – merv Apr 14 '21 at 16:51
  • 7
    If you are using conda the best option is to use `conda env export > environment.yml`, commit your environment.yml and then `conda env create -f environment.yml`. – Playing With BI Nov 03 '21 at 13:39
  • @PlayingWithBI, the OP is asking how to create a virtual environment on a machine which only has `pip` installed, not `conda`. – rmwenz Aug 07 '22 at 20:20
  • 1
    This answer is amazing, address every problem in a single post. I was having odd references and this solution helped me to solve it. – Quan Bui Oct 14 '22 at 03:47
  • Since pip uses double equal signs `==`, I recommend `conda list -e | sed 's/=/==/' | sed 's/=[^=]*$//' > requirements.txt`. This will convert `pandas=1.5.3=py311heda8569_0` to `pandas==1.5.3`. – datalifenyc Aug 20 '23 at 16:59
57

Problem

In a conda environment with simply calling

pip freeze

Output:

ipykernel @ file:///C:/ci/ipykernel_1607454116140/work/dist/ipykernel-5.3.4-py3-none-any.whl
ipython @ file:///D:/bld/ipython_1612487184680/work
...

Wanted format:

ipykernel==5.3.4
ipython==7.20.0
...

Solution

In an activated conda environment I had to use

pip list --format=freeze

to get the correct format for generating a requirements file for people who prefer to use pip with virtual environments.

Save to file:

pip list --format=freeze > requirements.txt
Pascal
  • 812
  • 7
  • 19
  • 9
    Works perfectly! Just use `pip list --format=freeze > requirements.txt` .To store package list as requirements.txt file in the project directory. – Pam Cesar Jun 01 '21 at 04:02
32

Following the discussion, I'd like to mention that you can actually see some separation of pip and conda roles.

pip is a standard package manager, it does one thing and does it well. requirements.txt can be generated in one environment and installed by pip in a new environment.

Now there is conda output: you rightfully capture their comment which says 'we generated this list of libraries to work with conda'. Note that python itself is in the conda list and (properly) not in requirements.txt. conda replicates own installation, that is why its list of libraries is longer, and has python itself.

pip produces a list of packages that were installed on top of standard library to make the package you wrote work. Hope it helps to distinguish between the two.

Also pipenv is a newer tool, that can do both virtual environment and package management for you.

Martijn Pieters
  • 1,048,767
  • 296
  • 4,058
  • 3,343
Evgeny
  • 4,173
  • 2
  • 19
  • 39
  • 16
    pipenv is still a tool on top of the Python base, just like pip is. It will not handle requirements *outside* of Python. That's what conda is for, to handle that larger picture, see [What is the difference between pip and conda?](//stackoverflow.com/a/20994790) – Martijn Pieters Jul 19 '18 at 10:17
  • 2
    While this answer provides some additional useful information, it does not provide actual commands to use with conda and pip. – Rich Lysakowski PhD Mar 23 '20 at 03:29
24

Just in case someone is looking to generate requirements.txt from an existing project in conda, use following

  • Go to your project environment conda activate <env_name>

  • conda list gives you list of packages used for the environment

  • conda list -e > requirements.txt save all the info about packages to your folder

  • conda env export > <env_name>.yml

  • pip freeze

Aishwat Singh
  • 4,331
  • 2
  • 26
  • 48
  • 7
    The OP wanted a usable requirements.txt file. How does this procedure help them get that? – mabraham Feb 27 '20 at 09:13
  • 5
    This answer is very useful because it provides the exact package list and commands needed to create the yaml file to recreate the conda environment, not just the list of pip packages. – Rich Lysakowski PhD Mar 23 '20 at 09:27
  • Aishwat thanks for sharing. can you explain how does this magic happen? – SKSKSKSK Dec 02 '20 at 10:28
5

As mentioned in the comments above, the correct full answer is:

pip list --format=freeze > requirements.txt
MattG
  • 5,589
  • 5
  • 36
  • 52
3

activate the conda env
conda activate flask-test

get the path of the conda env and copy it
conda list

append the copied path with lib\site-packages and use it in pip with --path option
pip freeze --path C:\Users\username\Miniconda3\envs\flask-test\lib\site-packages > requirements.txt on Linux the path is like /home/username/miniconda3/envs/flask-app/lib/python3.8/site-packages/

Edgar Manukyan
  • 1,153
  • 10
  • 21
0

I made some simple python script to convert conda requirements.txt for pip3. Just copy below code.

f = open(r"requirements.txt", "r").read()
f1 = f.split("\n")
for line in f1:  
    if "=" in line:
       a = line.split("=")[0:-1]
       print(a[0]+"==" + a[1])

then just copy the printed output;

Sample output:

_libgcc_mutex==0.1
_openmp_mutex==4.5
blas==1.0
brotli==1.0.9
bzip2==1.0.8
tensordt
  • 1