Python reload pip doesn't work

Question

I am currently trying to upgrade pip and then install paramiko inside a python script because the script itself makes use of paramiko. Below is a simple case of what I am trying to accomplish

import pip

try:
        import paramiko
except ImportError:
        pip.main(["install", "--upgrade", "pip"])
        reload(pip)
        pip.main(["install", "--user", "paramiko"])
        import paramiko

ssh = paramiko.SSHClient()

I am running this script in an Ubuntu 16.04 VM with python 2.7 which doesn't have paramiko and pip version 8.1.1. For the output of the pip upgrade after running the script, I get:

Collecting pip Downloading pip-9.0.1-py2.py3-none-any.whl (1.3MB) 100% |████████████████████████████████| 1.3MB 712kB/s Installing collected packages: pip Successfully installed pip-8.1.1 You are using pip version 8.1.1, however version 9.0.1 is available. You should consider upgrading via the 'pip install --upgrade pip' command.

After that, it tries to install paramiko as expected, but it fails because it basically can't install the Cryptography dependency that paramiko uses. However, if it was using the upgraded pip, this would not be a problem. I have reverted the VM back to a previous snapshot to manually install paramiko by doing

pip install --upgrade pip
pip install --user paramiko

in the shell and it works, but I need to be able to perform this in a script.

Note, I also reverted the VM to a previous snapshot and tried installing paramiko with the above command in the shell, but without performing the pip upgrade first, and as expected I get the same error as what the script gives when trying to install paramiko. Now I have done a bit of testing by trying to having this in the script:

pip.main(["install", "--upgrade", "pip"])
reload(pip)
pip.main(["install", "--upgrade", "pip"])
reload(pip)

and as for the output I get this:

Collecting pip
Downloading pip-9.0.1-py2.py3-none-any.whl (1.3MB)

100% |████████████████████████████████| 1.3MB 712kB/s Installing collected packages: pip Successfully installed pip-8.1.1 You are using pip version 8.1.1, however version 9.0.1 is available. 

You should consider upgrading via the 'pip install --upgrade pip' command.

Collecting pip
Using cached pip-9.0.1-py2.py3-none-any.whl

Installing collected packages: pip Successfully installed pip-8.1.1

You are using pip version 8.1.1, however version 9.0.1 is available.

You should consider upgrading via the 'pip install --upgrade pip' command.

Notice how it says using cached pip-9.0.1 at the second pip upgrade. Does this mean after the reload, the script is using pip 9.0.1? If so then why is the paramiko install not using the upgraded pip? How can I get the paramiko to use the upgraded pip?

Answer 1

Do you try to remove the pip package from the sys.modules array?

import pip
import sys
import site

print ('PIP Version: ' + pip.__version__)

pip.main(["install", "--upgrade", "pip"])

user_path  = site.getusersitepackages()
sytem_path = site.getsitepackages()

sys.path = [user_path] + system_path + sys.path

pips = [module for module in sys.modules if module.startswith('pip')]
for pip_module in pips:
    sys.modules.pop(pip_module)

del pip

import pip
print ('PIP Version: ' + pip.__version__)

Runinng on my machine with pip 8.1.1 installed, the output I get was:

 begnini@coffee-machine:~# python update.py
 PIP Version: 8.1.1
 Collecting pip
  Downloading pip-9.0.1-py2.py3-none-any.whl (1.3MB)
    100% |████████████████████████████████| 1.3MB 651kB/s
 Installing collected packages: pip
  Found existing installation: pip 8.1.1
    Uninstalling pip-8.1.1:
      Successfully uninstalled pip-8.1.1
 Successfully installed pip-9.0.1
 PIP Version: 9.0.1

EDIT: Like we discussed, the problem was with pip trying to search first in the main python lib directory instead /usr/local/lib dir. To fix this, I added the local lib dir in the first position of the system path array. With this modification, the bug should be fixed.

EDIT 2: Added site paths instead use hardcoded dirs.

Answer 2

So after some digging around, it looks like this is a pip bug that has been patched. Unfortunately, I may not always have control over the machine that will run this script (i.e. can't apply the patch) so my solution was just to use os.execl() to restart the script whenever the second import for paramiko in the except section fails.

As for the bug, in summary, after a pip upgrade, the wrong version was reported, but pip was upgraded. So if the script was ran again, the new version of pip was used which allowed paramiko to be installed properly, hence the solution to restart the script.

For guidance on using os.execl() since python documentation is not very clear on it, follow this link

Sample of how my code looks now:

import pip
import sys
import os

try:
        import paramiko
except ImportError:
        pip.main(["install", "--upgrade", "pip"])
        pip.main(["install", "--user", "paramiko"])
        try:
            import paramiko
        except ImportError:
            os.execl(sys.executable, 'python', __file__, *sys.argv[1:])

ssh = paramiko.SSHClient()