Pip install in Virtualenv working, but not working with `-r requirements.txt`

Question

I am hitting a unique version of the well documented Virtualenv-Pip bug discussed in answers like here and here. In these cases the issue was pip was installing in a global folder, in some cases because the shebang in one of the pip scripts was wrong, in others because the line 42 in the bin/activate script was wrong, but it always installed globally.

My issue is different: within my virtual environment pip install <package> works as desired (installing locally), but pip install -r requirements.txt installs into the void. For example:

pip install scipy

then looking at the contents of /home/user/software/project/venv/lib/python3.6/site-packages i get the result:

easy_install.py  
numpy  
numpy-1.16.1.dist-info  
pip  
pip-19.0.2.dist-info  
pkg_resources  
__pycache__  
scipy  
scipy-1.2.1.dist-info  
setuptools  
setuptools-40.8.0.dist-info  
wheel  
wheel-0.33.0.dist-info

However, when in the same virtual environment, I run pip install -r requirements.txt, where the requirements.txt asks to install numpy, Cython, sklearn, matplotlib, and argparse, and which outputs:

Collecting Cython (from -r requirements.txt (line 1))                                                                                                                                                              
  Downloading https://files.pythonhosted.org/packages/16/98/49aa24054e99e9c7734e49d6996662f547e4e2faae0051d35fbbc461afa4/Cython-0.29.5-cp36-cp36m-manylinux1_x86_64.whl (2.1MB)                                    
    100% |████████████████████████████████| 2.1MB 23.6MB/s                                                                                                                                                         
Collecting numpy (from -r requirements.txt (line 2))                                                                                                                                                               
  Using cached https://files.pythonhosted.org/packages/f5/bf/4981bcbee43934f0adb8f764a1e70ab0ee5a448f6505bd04a87a2fda2a8b/numpy-1.16.1-cp36-cp36m-manylinux1_x86_64.whl                                            
Collecting sklearn (from -r requirements.txt (line 3))                                                                                                                                                             
  Downloading https://files.pythonhosted.org/packages/1e/7a/dbb3be0ce9bd5c8b7e3d87328e79063f8b263b2b1bfa4774cb1147bfcd3f/sklearn-0.0.tar.gz
Collecting matplotlib (from -r requirements.txt (line 4))
  Downloading https://files.pythonhosted.org/packages/71/07/16d781df15be30df4acfd536c479268f1208b2dfbc91e9ca5d92c9caf673/matplotlib-3.0.2-cp36-cp36m-manylinux1_x86_64.whl (12.9MB)
    100% |████████████████████████████████| 12.9MB 6.3MB/s
Collecting argparse (from -r requirements.txt (line 5))
  Downloading https://files.pythonhosted.org/packages/f2/94/3af39d34be01a24a6e65433d19e107099374224905f1e0cc6bbe1fd22a2f/argparse-1.4.0-py2.py3-none-any.whl

This results in no new additions to the local site-packages. What's more when I try to see where they installed, there are no site-packages directories in /usr/lib/python2.7, /usr/local/lib/python2.7, /usr/lib/python3, /usr/lib/python3.6, usr/lib/python3.7, ~/.local/lib/python3.6/, or ~/.local/lib/python2.7/.

To verify this, when I run: find /usr/ -name "Cython" I get an empty result.

Any advice? I can write a bash script that literally calls pip install instead, but that seems so sloppy, and I feel like I should fix this before it becomes worse.

NOTES:

My machine is on 18.04.1, which has python3.6.7 as python3 and python 2.7.15 as python2 installed as machine defaults, I installed pip for both with sudo apt install python-pip and sudo apt install python3-pip, and this virtual environment was created with virtualenv -p python3 venv

My requirements.txt:

Cython
numpy
sklearn
matplotlib
torch>=0.4.1
torchvision>=0.2.1
argparse
quadprog

Full output from pip install -r requirements.txt

Also, the complete `pip` output in addition to the Collecting? — Chris Larson, Feb 14 '19 at 03:40
Add what happens when you manually `pip` install the rest? `Cython`, `sklearn`, `matplotlib`, and `argparse`? — Chris Larson, Feb 14 '19 at 03:42
adding my `requirements.txt` and the full `pip` output. When I manually install the rest they just work normally and are in the correct site-packages area. — physincubus, Feb 14 '19 at 18:46
And when you run `find /home/user/software/project/ -name sklearn` for example? (And for the record, `argparse` is built in with Python >=3.2.) — Chris Larson, Feb 14 '19 at 20:50
@ChrisLarson yes I mean ubuntu, and yes that find command returns empty — physincubus, Feb 14 '19 at 23:10
Can you try nuking your `venv` directory and then trying `python3 -m venv venv; . venv/bin/activate` from your project directory? I'm curious if something is messed up in the `venv` or with your `virtualenv`. — Chris Larson, Feb 14 '19 at 23:28

score 0 · Answer 1 · answered Feb 18 '19 at 02:15

So this is a partial answer, but I figured out at least the problem even if I don't entirely understand why. When I changed:

virtualenv -p python3 venv

to

virtualenv -p /usr/lib/python3.6 venv .

pip install -r requirements.txt worked, yielding the correct site-packages.

Again, I do not know why this worked, but it did. I just tried it again with a different python version with less luck though, so I'm not entirely sure if this is a universal fix. I will edit this answer when I have a better explanation.

Pip install in Virtualenv working, but *not* working with `-r requirements.txt`

1 Answers1

Pip install in Virtualenv working, but not working with `-r requirements.txt`