Virtualenv, pip-sync and "ERROR: Cannot uninstall"

Question

I'm on Ubuntu 20.04 desktop. I cannot recall if I had to install python3-pip manually here (have had to on other Ubuntu machines), but I do have the system one:

$ apt-show-versions -r python3-pip
python3-pip:all/focal-updates 20.0.2-5ubuntu1.6 uptodate

... and then:

sudo -H pip3 install --upgrade pip
sudo -H pip3 install pip-tools pipdeptree
sudo -H pip3 install virtualenv

Then, try to set up virtualenv - from my home directory:

virtualenv venv_mpl

Then, I place following contents in venv_mpl/requirements.in:

plotly
seaborn
scikit-learn

Then, I enable the virtualenv, and try to install packages:

$ source venv_mpl/bin/activate
(venv_mpl) $ pip-compile venv_mpl/requirements.in
...
(venv_mpl) $ pip-sync venv_mpl/requirements.txt
Found existing installation: apturl 0.5.2
ERROR: Cannot uninstall 'apturl'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.
Traceback (most recent call last):
...

Well, yes, indeed I do have apturl installed previously:

$ apt-show-versions -r apt.*url
apturl:amd64/focal 0.5.2ubuntu19 uptodate
apturl-common:amd64/focal 0.5.2ubuntu19 uptodate

... but then again, wasn't it supposed to be so, that if I use virtualenv, then I'd not have this kind of problems? At least, that is the reading I got from pip cannot uninstall <package>: "It is a distutils installed project" :

Alternatively, depending on your needs, it may be more productive to not use your system Python and/or its global environment but create a private Python installation and/or environment. There are many options here including virtualenv ...

So, is there anything I can do, to have the pip packages collected by pip-compile from the above requirements.in installed in the virtualenv, created as noted above?

EDIT: If I attempt to remove the system apturl, it also tries to remove a critical desktop component, in this case, ubuntu-mate-core:

$ sudo apt remove --dry-run apturl
[sudo] password for administrator: 
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages will be REMOVED:
  apturl ubuntu-mate-core
0 upgraded, 0 newly installed, 2 to remove and 21 not upgraded.
Remv ubuntu-mate-core [1.263]
Remv apturl [0.5.2ubuntu19]

Please [don't `sudo` for `pip` installs](https://stackoverflow.com/questions/29310688/sudo-pip-install-vs-pip-install-user); if you want to make sure you're using the right Python, use `/path/to/python -m pip`. Installation, by design, runs arbitrary Python code from `setup.py` (for packages that use it). Also, you shouldn't have to apt-install pip; it's supposed to come with the Python distribution. — Karl Knechtel, Dec 15 '21 at 03:46
Thanks, @KarlKnechtel - I think I have only one Python3 (`/usr/bin/python3`), and as far as I can see `venv_mpl/bin/python3` symlinks to that one. Only reason I used sudo, was because I wanted to install those `pip` package as "system" ones (similar to how `apt install python3-matplotlib` would have been); so far I haven't found better instructions on how to do that, than using `sudo`? In any case, I still don't have an idea, about what I should do, to get this virtualenv populated with the required pip packages, and running? — sdbbs, Dec 15 '21 at 03:51
Generally the recommendation is "don't install packages as system packages; there are tools that can easily automate installing packages into new virtual environments". — Karl Knechtel, Dec 15 '21 at 03:53
Thanks, @KarlKnechtel - but that still doesn't explain what I should do to get this running; "... there are tools..." - sure, which tools? Say, I don't do `sudo -H pip3 install virtualenv` - how am I supposed to run virtualenv then, so I create the virtual environment? — sdbbs, Dec 15 '21 at 03:57
Virtual environment creation is provided by the `venv` standard library module [since 3.3](https://docs.python.org/3/library/venv.html). You again shouldn't need to install anything. — Karl Knechtel, Dec 15 '21 at 04:00
Thanks @KarlKnechtel - I had no idea about `venv`; so I repeated the procedure following your advice, and now everything works (posted this as an answer). — sdbbs, Dec 15 '21 at 04:29

sdbbs · Answer 1 · 2021-12-15T13:14:53.527

As @KarlKnechtel pointed out, sudo should not be used with pip, so I tried to do it that way - first, restore previous state (from the changes done in OP):

# answer Y to all questions at next command
sudo -H pip3 uninstall virtualenv pip-tools pipdeptree
rm -rf venv_mpl

Now, since this is currently Python 3.8.10, I can use the venv module, which is standard since 3.3 - however, I first need to do this (error message recommended python3.8-venv, but python3-venv is probably better):

sudo apt install python3-venv

... and then, I can create virtualenv:

python3 -m venv venv_mpl

Now, let's proceed in the venv, and run commands as normal user:

$ source venv_mpl/bin/activate

(venv_mpl) $ pip3 install --upgrade pip
...
Successfully installed pip-21.3.1

(venv_mpl) $ pip3 install pip-tools pipdeptree
Collecting pip-tools
...
Successfully installed click-8.0.3 pep517-0.12.0 pip-tools-6.4.0 pipdeptree-2.2.0 tomli-2.0.0 wheel-0.37.0

(venv_mpl) $ cat > venv_mpl/requirements.in <<'EOF'
plotly
seaborn
scikit-learn
EOF

(venv_mpl) $ pip-compile venv_mpl/requirements.in
...

(venv_mpl) $ pip-sync venv_mpl/requirements.txt
Found existing installation: pipdeptree 2.2.0
Uninstalling pipdeptree-2.2.0:
  Successfully uninstalled pipdeptree-2.2.0
Collecting cycler==0.11.0
  Using cached cycler-0.11.0-py3-none-any.whl (6.4 kB)
Collecting fonttools==4.28.3
  Using cached fonttools-4.28.3-py3-none-any.whl (884 kB)
Collecting joblib==1.1.0
  Using cached joblib-1.1.0-py2.py3-none-any.whl (306 kB)
Collecting kiwisolver==1.3.2
  Using cached kiwisolver-1.3.2-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl (1.2 MB)
Collecting matplotlib==3.5.1
  Using cached matplotlib-3.5.1-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl (11.3 MB)
Collecting numpy==1.21.4
  Using cached numpy-1.21.4-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (15.7 MB)
Collecting packaging==21.3
  Using cached packaging-21.3-py3-none-any.whl (40 kB)
Collecting pandas==1.3.5
  Using cached pandas-1.3.5-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.5 MB)
Collecting pillow==8.4.0
  Using cached Pillow-8.4.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.1 MB)
Collecting plotly==5.4.0
  Using cached plotly-5.4.0-py2.py3-none-any.whl (25.3 MB)
Collecting pyparsing==3.0.6
  Using cached pyparsing-3.0.6-py3-none-any.whl (97 kB)
Collecting python-dateutil==2.8.2
  Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting pytz==2021.3
  Using cached pytz-2021.3-py2.py3-none-any.whl (503 kB)
Collecting scikit-learn==1.0.1
  Using cached scikit_learn-1.0.1-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (25.9 MB)
Collecting scipy==1.7.3
  Using cached scipy-1.7.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (39.3 MB)
Collecting seaborn==0.11.2
  Using cached seaborn-0.11.2-py3-none-any.whl (292 kB)
Collecting six==1.16.0
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting tenacity==8.0.1
  Using cached tenacity-8.0.1-py3-none-any.whl (24 kB)
Collecting threadpoolctl==3.0.0
  Using cached threadpoolctl-3.0.0-py3-none-any.whl (14 kB)
Installing collected packages: six, pyparsing, pytz, python-dateutil, pillow, packaging, numpy, kiwisolver, fonttools, cycler, threadpoolctl, tenacity, scipy, pandas, matplotlib, joblib, seaborn, scikit-learn, plotly
Successfully installed cycler-0.11.0 fonttools-4.28.3 joblib-1.1.0 kiwisolver-1.3.2 matplotlib-3.5.1 numpy-1.21.4 packaging-21.3 pandas-1.3.5 pillow-8.4.0 plotly-5.4.0 pyparsing-3.0.6 python-dateutil-2.8.2 pytz-2021.3 scikit-learn-1.0.1 scipy-1.7.3 seaborn-0.11.2 six-1.16.0 tenacity-8.0.1 threadpoolctl-3.0.0

Well, it did succeed here, without bumping into the apturl issue - great!

I find it a bit weird, though, that pipdeptree had to be removed for this to succeed (EDIT: it turns out, pipdeptree also has to be added to requirements.in, then it gets installed as all others there - strange that pip-tools does not need this, then again, pip-tools is the one doing the parsing of the requirements file; possibly related: note that sometimes "The following packages are considered to be unsafe in a requirements file:" gets printed, there is some discussion on that here Reasoning behind unsafe packages · Issue #522 · jazzband/pip-tools · GitHub), but hey - at least the procedure passed, and the right packages got installed, so ... great!

EDIT2: if when trying to run a matplotlib script from this virtual environment, Python responds with "UserWarning: Matplotlib is currently using agg, which is a non-GUI backend, so cannot show the figure.", one needs to install the system sudo apt install python3-tk (see "UserWarning: Matplotlib is currently using agg, which is a non-GUI backend, so cannot show the figure." when plotting figure with pyplot on Pycharm)

Virtualenv, pip-sync and "ERROR: Cannot uninstall"

1 Answers1