0

There are two similar posts (post1, post2) on SO that is asking about a similar situation, however, after following all the steps in those questions, I still can't figure out why the sklearn versions are mismatching.

I'm working in a cloud hosted Jupyter notebook and I'm presume that the runtime is from some image of which I'm not sure how it is set up.

Below are the outputs to ensure that:

  • The pip that is used is associated with the right python.
  • The pip installed sklearn is indeed of a specific version.
  • The output from sklearn.__version__.

Input:

import sklearn

# Ensures that the proper bin is used:
print("echo $PATH output: ")
!echo $PATH
print()

# Shows that the scikit-learn version is 1.0.2
print("Attempting to upgrade scikit-learn with the correct pip: ")
!python3 -m pip install scikit-learn --upgrade
print()

# Check if the pip used is the one associated with the Python I'm using:
print("which pip output: ")
!which pip
!pip --version
!python3 -m pip --version
print()

# Doesn't matter which Python I call, they all point to 3.7
print("All python points to 3.7: ")
!which python3
!ls -lah /opt/conda/bin | grep python
print()

# Shows scikit-learn version 1.0.2
print("pip list grep output: ")
!pip list | grep scikit
print()

# Shows scikit-learn version 1.0.2
print("pip show output: ")
!pip show scikit-learn
print()

# Shows scikit-learn version 0.22.2
print("sklearn.__version__ output: ")
print(sklearn.__version__)
print()

# sklearn is in the correct python3.7 sub-directory
print("sklearn.__path__ output: ")
print(sklearn.__path__)

Output:

echo $PATH output: 
/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

Attempting to upgrade scikit-learn with the correct pip: 
Requirement already satisfied: scikit-learn in /opt/conda/lib/python3.7/site-packages (1.0.2)
Requirement already satisfied: threadpoolctl>=2.0.0 in /opt/conda/lib/python3.7/site-packages (from scikit-learn) (3.1.0)
Requirement already satisfied: numpy>=1.14.6 in /opt/conda/lib/python3.7/site-packages (from scikit-learn) (1.18.4)
Requirement already satisfied: joblib>=0.11 in /opt/conda/lib/python3.7/site-packages (from scikit-learn) (0.14.1)
Requirement already satisfied: scipy>=1.1.0 in /opt/conda/lib/python3.7/site-packages (from scikit-learn) (1.4.1)

which pip output: 
/opt/conda/bin/pip
pip 23.0.1 from /opt/conda/lib/python3.7/site-packages/pip (python 3.7)
pip 23.0.1 from /opt/conda/lib/python3.7/site-packages/pip (python 3.7)

All python points to 3.7: 
/opt/conda/bin/python3
-rwxr-xr-x  1 jovyan users  230 Apr  8 22:30 ipython
-rwxr-xr-x  1 jovyan users  230 Apr  8 22:30 ipython3
lrwxrwxrwx  1 jovyan users    9 May  3  2020 python -> python3.7
lrwxrwxrwx  1 jovyan users    9 May  3  2020 python3 -> python3.7
-rwxrwxr-x  1 jovyan users  13M May  3  2020 python3.7
lrwxrwxrwx  1 jovyan users   17 May  3  2020 python3.7-config -> python3.7m-config
lrwxrwxrwx  1 jovyan users    9 May  3  2020 python3.7m -> python3.7
-rwxrwxr-x  1 jovyan users 3.3K May  3  2020 python3.7m-config
lrwxrwxrwx  1 jovyan users   17 May  3  2020 python3-config -> python3.7m-config

pip list grep output: 
scikit-image           0.16.2
scikit-learn           1.0.2

pip show output: 
Name: scikit-learn
Version: 1.0.2
Summary: A set of python modules for machine learning and data mining
Home-page: http://scikit-learn.org
Author: 
Author-email: 
License: new BSD
Location: /opt/conda/lib/python3.7/site-packages
Requires: joblib, numpy, scipy, threadpoolctl
Required-by: sklearn

sklearn.__version__ output: 
0.22.2.post1

sklearn.__path__ output: 
['/opt/conda/lib/python3.7/site-packages/sklearn']

Troubleshooting Attempted:

  • Verified the correct pip is used.
  • pip upgrade sklearn
  • Checked that all python versions points to the same one (i.e. Python3.7)
  • Compared the sklearn.__path__ matches the location where pip installs into
qwerty
  • 101
  • 1
  • 9
  • How are you running that python script? – Ben Reiniger Apr 09 '23 at 02:54
  • Hey Ben, they’re all ran in a single Jupiter notebook cell. – qwerty Apr 09 '23 at 06:01
  • And how do you initialize Jupyter? – Ben Reiniger Apr 12 '23 at 14:47
  • The notebook is initiative in a cloud environment, and after some digging via Jupyter's `!` function calls, it seems that Python 3.7 is installed via conda and all other packages are installed via pip under this Python3.7. The notebook is being used for an assignment thus everything is configured by the professor. All I do is to click a link and the notebook gets spun up. – qwerty Apr 13 '23 at 15:52
  • However, I think I may have figured out what is going on. After `pip install scikit-learn` I should've restarted the kernel and re-import sklearn. This way the versions from `pip show scikit-learn` and `print(sklearn.__version__)` will match. – qwerty Apr 13 '23 at 15:55

1 Answers1

1

After some digging, the following procedure resolves the version mismatch.

  1. Pip install the desired scikit-learn version !python3 -m pip install --force-reinstall "scikit-learn==1.0.2"

  2. Restart Jupyter notebok's kernel.

  3. Import sklearn and verify versions:

!pip show scikit-learn      # Should show v1.0.2

import sklearn
print(sklearn.__version__)  # Should also show v1.0.2

Moral of the story:

Restart the kernel and reimport after any update or installation for the updated library to be reflected in the ipynb.

qwerty
  • 101
  • 1
  • 9