Pip install upgrade unable to remove temp files

Question

In the course of maintaining a CLI utility, I want to add an update action that will grab the latest version of that package from PyPI and upgrade the existing installation.

$ cli -V
1.0.23

$ cli update
// many lines of pip spam

$ cli -V
1.0.24  // or etc

This is working perfectly on all machines that have Python installed system-wide (in C:\Python36 or similar), but machines that have Python installed as a user (in C:\users\username\AppData\Local\Programs\Python\Python36) receive this error as the old version is uninstalled:

Could not install packages due to an EnvironmentError: [WinError 5] Access is denied: 'C:\\Users\\username\\AppData\\Local\\Temp\\pip-uninstall-f5a7rk2y\\cli.exe'
Consider using the `--user` option or check the permissions.

I had assumed that this is due to the fact that the cli.exe called out in the error text is currently running when pip tries to remove it, however the path here is not to %LOCALAPPDATA%\Programs\Python\Python36\Scripts where that exe lives, but instead to %TEMP%. How is it allowed to move the file there, but not remove it once it's there?

including --user in the install args as recommended by the error message does not (contrary to the indication of an earlier edit of this question) resolve the issue, but moving the cli executable elsewhere does.

I'm hoping for an answer that:

1. Explains the underlying issue of failing to delete the executable from the TEMP directory, and...
2. Provides a solution to the issue, either to bypass the permissions error, or to query to see if this package is installed as a user so the code can add --user to the args.

While the question is fairly general, a MCVE is below:

def update(piphost):
    args = ['pip', 'install',
        '--index-url', piphost,
        '-U', 'cli']
    subprocess.check_call(args)

update('https://mypypiserver:8001')

Answer 1

As originally surmised, the issue here was trying to delete a running executable. Windows isn't a fan of that sort of nonsense, and throws PermissionErrors when you try. Curiously though, you can definitely rename a running executable, and in fact several questions from different tags use this fact to allow an apparent change to a running executable.

This also explains why the executable appeared to be running from %LOCALAPPDATA%\Programs\Python\Python36\Scripts but failing to delete from %TEMP%. It has been renamed (moved) to the %TEMP% folder during execution (which is legal) and then pip attempts to remove that directory, also removing that file (which is illegal).

The implementation goes like so:

1. Rename the current executable (Path(sys.argv[0]).with_suffix('.exe'))
2. pip install to update the package
3. Add logic to your entrypoint that deletes the renamed executable if it exists.

import click  # I'm using click for my CLI, but YMMV
from pathlib import Path
from sys import argv

def entrypoint():
    # setup.py's console_scripts points cli.exe to here

    tmp_exe_path = Path(argv[0]).with_suffix('.tmp')
    try:
        tmp_exe_path.unlink()
    except FileNotFoundError:
        pass
    return cli_root

@click.group()
def cli_root():
    pass

def update(pip_host):

    exe_path = Path(argv[0])
    tmp_exe_path = exe_path.with_suffix('.tmp')
    handle_renames = False
    if exe_path.with_suffix('.exe').exists():
        # we're running on Windows, so we have to deal with this tomfoolery.
        handle_renames = True
        exe_path.rename(tmp_exe_path)
    args = ['pip', 'install',
        '--index-url', piphost,
        '-U', 'cli']
    try:
        subprocess.check_call(args)
    except Exception:  # in real code you should probably break these out to handle stuff
        if handle_renames:
            tmp_exe_path.rename(exe_path)  # undo the rename if we haven't updated

@cli_root.command('update')
@click.option("--host", default='https://mypypiserver:8001')
def cli_update(host):
    update(host)

Answer 2

Great solution provided by the previous commenter: Pip install upgrade unable to remove temp files by https://stackoverflow.com/users/3058609/adam-smith

I want to add to remarks that made the code work for me (using python 3.8):

• Missing parentheses for the return of the entrypoint function (however, the function I'm pointing to is not decorated with @click.group() so not sure if that's the reason)

def entrypoint():
    # setup.py's console_scripts points cli.exe to here

    tmp_exe_path = Path(argv[0]).with_suffix('.tmp')
    try:
        tmp_exe_path.unlink()
    except FileNotFoundError:
        pass
>>> return cli_root() <<<

• Missing with_suffix('.exe') when attempting the rename in the update function. If I use the original code I get FileNotFoundError: [WinError 2] The system cannot find the file specified: '..\..\..\updater' -> '..\..\..\updater.tmp'

def update(pip_host):

    exe_path = Path(argv[0])
    tmp_exe_path = exe_path.with_suffix('.tmp')
    handle_renames = False
    if exe_path.with_suffix('.exe').exists():
        # we're running on Windows, so we have to deal with this tomfoolery.
        handle_renames = True
>>>     exe_path.with_suffix('.exe').rename(tmp_exe_path)    <<<
        
        ...