0

Context

On a provided Linux-server without root access I am bound to preinstalled Python-packages (e.g. cannot upgrade them).
However, I am able to install a package directly from a wheel (.whl) by using pip3 install /path/package_name --user which will install the package to a site-package-folder preserved to my user.

In my special case I want to upgrade the scikit-learn-package. This package is already preinstalled and I cannot upgrade it (root access missing), however, I can install the newer version in the --user-folder.

In the filesystem I can see that both installations are now present in their respective folders. Both paths are known to python (checked by using sys.path).

Question/Problem

When I import scikit-learn via import sklearn and print the version (sklearn.__version__), I'll always end up with the preinstalled version and not the new one I installed in my --user-directory.

Given two installations of the same module with a differing version: How can I define in Python during the import which module/version to load?

The premise is that I cannot disable/uninstall the old version (root access again..).

Markus
  • 2,265
  • 5
  • 28
  • 54
  • The user is always the same here? – 9769953 Sep 03 '20 at 17:31
  • 1
    Did you verify that `pip3` is the command that fits the `python(3)` command? That is, you didn't accidentally install the upgraded scikit-learn for one Python, then try to import it in another Python version. – 9769953 Sep 03 '20 at 17:32
  • 1
    Coud you check what `sys.path` is on the Python prompt? That is, what is the order of paths? – 9769953 Sep 03 '20 at 17:40
  • The user stays the same and pip3 is connected to Python3 (which can also been seen when I import other modules installed to the `--user`. But regarding the order: The --user-path comes after the other one - I will switch them and take a look at the result (seems very promising! :) ) – Markus Sep 03 '20 at 18:19
  • @00: Thanks for the input! Maybe I missunderstood you (because I changed `sys.path` and this did not change anything), but changing the `$PYTHONPATH` which initializes `sys.path` did the trick! Thanks a lot :) – Markus Sep 03 '20 at 18:57

1 Answers1

0

Thanks to @0 0 I was able to figure out the solution as well as a workaround:

Solution

When I compared sys.path with the defined PYTHONPATH in the shell (the PYTHONPATH initializes sys.path like answered here on SO) I saw that the PYTHONPATH did not know about the --user-folder (even though sys.path knew - this puzzles me a bit).

Thus, I added to my .bashrc:

cd # Go to the user folder where .bashrc is stored

nano .bashrc
export PYTHONPATH=$(python3 -m site --user-site)$PYTHONPATH

python3 -m site --user-site identifies the --user-folder pip uses. In order to load the module first, put it before the current PYTHONPATH.

When now loading sklearn in Python I get the correct version.

Workaround

For the sake of completeness: One can also just rename the doubled package and import that.

cd $(python3 -m site --user-site) # Go to the folder pip installed the package while using --user

mv package_name package_name_new # Rename it

Then in Python load it this way:

import package_name_new

NOTE: This workaround will cause trouble if you have dependencies relying on it! Thus, better use the correct solution.

Markus
  • 2,265
  • 5
  • 28
  • 54
  • 1
    Usually, PYTHONPATH is blank (or unset); I'm not sure if, in that case, the user installed module path takes precedence over the system installed module path. It seems that here, PYTHONPATH was the problematic part, because it was manually altered. – 9769953 Sep 03 '20 at 23:30
  • 1
    According to [PEP 370](https://www.python.org/dev/peps/pep-0370/): "The user site directory is added before the system site directories but after Python's search paths and PYTHONPATH. This setup allows the user to install a different version of a package than the system administrator but it prevents the user from accidentally overwriting a stdlib module. Stdlib modules can still be overwritten with PYTHONPATH.". That reads as if the user-installed package should take precedence. – 9769953 Sep 04 '20 at 07:08
  • An alternative would be to use a virtual environment instead. – 9769953 Sep 04 '20 at 07:10
  • @00: _"[...] PYTHONPATH was the problematic part, because it was manually altered"_ - Yeah, it seems that the admin choosed for whatever reason to set this manually and did not know about the drawbacks either. – Markus Sep 04 '20 at 09:13