10

I know this topic has been beat to death but I have not been able to find a solution to the problem I'm having on SO or elsewhere, so I suspect that there may be a bug somewhere in my system.

I am on an older RHEL 6 platform with Python 3.4. I am developing an application that will run on this platform that uses Qt. I've installed all of the relevant libraries via yum (e.g. qt-devel, pyqt4-devel, etc.) and now want to install my application package as an "editable" package using pip install -e mypkg. I also have a couple of dependency requirements that are not on yum and must be installed via pip.

What I would like to do is create a virtualenv that "inherits" the system packages installed via yum but allows me to pip install my own packages into a virtualenv directory in my home directory.

From my Googling it looks like the best way to do this is to create a virtual env with the system's site packages directory:

$ python3 -m venv --system-site-packages ~/venv

However, when I try to install a package to this virtualenv's site-packages directory, it attempts to install it under /usr/lib and I get a Permission denied error.

So it appears that the --system-site-packages option makes my virtualenv completely share the site-packages directory from my system instead of using it as a "base", where further packages can be layered on top.

This answer states that using pip install -I should do what I want, but that does not appear to be the case:

(venv) $ pip3 install -I bitstring
...
error: could not create '/usr/lib/python3.4/site-packages/bitstring.py': Permission denied
gpanders
  • 1,301
  • 2
  • 15
  • 23
  • Did you activated the virtual env before trying to use pip again? – marcanuy Apr 09 '19 at 19:42
  • 1
    @marcanuy Yes, I attempted to indicate that with the `(venv)` in my prompt string – gpanders Apr 09 '19 at 20:07
  • Are you certain you need to configure the virtual environment to use system packages? I seem to recall that packages installed via `yum` or `apt` are available in virtual envs even without enabling system packages (much to my surprise at the time). I would at least test. – jpmc26 Apr 09 '19 at 22:12
  • @jpmc26 That does seem reasonable, but alas I don't think that is the case: `>>> import sys; print(sys.path)` shows `['', '/usr/lib64/python34.zip', '/usr/lib/python3.4', '/usr/lib64/python3.4/plat-linux', '/usr/lib64/python3.4/lib-dynload', '$HOME/venv/lib64/python3.4/site-packages', '$HOME/venv/lib/python3.4/site-packages']`, which is missing `/usr/lib64/python3.4/site-packages` – gpanders Apr 09 '19 at 22:16
  • @GregAnders No, no. Just try importing the module. E.g., `from PyQt4 import QtGui`. If it doesn't work, it doesn't, but my recollection is that it does (at least in some cases). Perhaps it gets installed to one of the other locations in your list. – jpmc26 Apr 09 '19 at 22:19
  • 1
    @jpmc26 I did try it with numpy (also installed via yum) and got `ImportError: No module named 'numpy'` – gpanders Apr 09 '19 at 22:41

2 Answers2

17

Create the virtual environment without the --system-site-packages switch. After the environment was created go to the folder the environment was created in. It should have a file pyvenv.cfg. Edit this file. It has (among other text) a line

include-system-site-packages = false

Change this line to:

include-system-site-packages = true

Activate the environment. Module installations will now go to the virtual environment and the system site packages are visible too.

rfindeis
  • 461
  • 3
  • 15
3

With Python 3.8, it seems --system-site-packages work as expected:

python3 -m venv --system-site-packages myProject
cat myProject/pyvenv.cfg 

home = /usr/bin
include-system-site-packages = true
version = 3.8.5

After installation astroid, isort, wrapt, I got:

pip list -v
Package                Version              Location                                                Installer
---------------------- -------------------- ------------------------------------------------------- ---------
apturl                 0.5.2                /usr/lib/python3/dist-packages                                   
astroid                2.4.2                /home/to/no/MR/auto-gen/lib/python3.8/site-packages pip      
isort                  5.6.4                /home/to/no/MR/auto-gen/lib/python3.8/site-packages pip      
jedi                   0.15.2               /usr/lib/python3/dist-packages                                   
keyring                18.0.1               /usr/lib/python3/dist-packages                                   
wrapt                  1.12.1               /home/to/no/MR/auto-gen/lib/python3.8/site-packages pip

Already installed 'system' packages are taken from /usr/lib/python3/dist-packages while locally (venv) installed packages from: /home/to/no/MR/auto-gen/lib/python3.8/site-packages

xhudik
  • 2,414
  • 1
  • 21
  • 39