Removing and reinstalling Python-Sphinx on Windows

Question

I am trying to use Python (3.8) and Sphinx (3.3.1) to build a documentation in HTML. However, the sphinx-build command gives me the following error:

C:\Users\Me\Dropbox\Kuchen>sphinx-build -b html source build
Running Sphinx v3.3.1
loading pickled environment... done
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 1 source files that are out of date
updating environment: 0 added, 1 changed, 0 removed
reading sources... [100%] kaesekuchen
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [100%] kaesekuchen
generating indices... genindex done
writing additional pages... search done
copying static files... WARNING: Failed to copy a file in html_static_file: c:\users\me\appdata\local\programs\python\python38\lib\site-packages\sphinx\themes\basic\static/jquery-3.5.1.js: PermissionError(13, 'Permission denied')
WARNING: Failed to copy a file in html_static_file: c:\users\me\appdata\local\programs\python\python38\lib\site-packages\sphinx\themes\basic\static/jquery.js: PermissionError(13, 'Permission denied')
done
copying extra files... done
dumping search index in English (code: en)... done
dumping object inventory... done
build succeeded, 2 warnings.

However,

The HTML file kaesekuchen in build is not updated/changed.
The folder c:\users\me\appdata\local\programs\python\python38\lib\site-packages\sphinx does not exist.

The latter is my fault, because I deleted it in file explorer, but only because I encountered the exact same error before, and hoped that deleting and re-installing Sphinx would solve it.

Instead, the commands pip uninstall sphinx and subsequent pip install -U sphinx do not change anything in that folder, and the latter only gives the following upbeat output despite the following output:

Microsoft Windows [Version 10.0.18363.1198]
(c) 2019 Microsoft Corporation. All rights reserved.

C:\Users\me>pip uninstall sphinx
Found existing installation: Sphinx 3.3.1
Uninstalling Sphinx-3.3.1:
  Would remove:
    c:\users\me\appdata\local\programs\python\python38\lib\site-packages\sphinx-3.3.1.dist-info\*
    c:\users\me\appdata\local\programs\python\python38\lib\site-packages\sphinx\*
    c:\users\me\appdata\local\programs\python\python38\scripts\sphinx-apidoc.exe
    c:\users\me\appdata\local\programs\python\python38\scripts\sphinx-autogen.exe
    c:\users\me\appdata\local\programs\python\python38\scripts\sphinx-build.exe
    c:\users\me\appdata\local\programs\python\python38\scripts\sphinx-quickstart.exe
Proceed (y/n)? y
  Successfully uninstalled Sphinx-3.3.1

C:\Users\me>pip install -U sphinx
Collecting sphinx
  Using cached Sphinx-3.3.1-py3-none-any.whl (2.9 MB)
Requirement already satisfied, skipping upgrade: docutils>=0.12 in c:\users\me\appdata\local\programs\python\python38\lib\site-packages (from sphinx) (0.16)
Requirement already satisfied, skipping upgrade: sphinxcontrib-serializinghtml in c:\users\me\appdata\local\programs\python\python38\lib\site-packages (from sphinx) (1.1.4)
Requirement already satisfied, skipping upgrade: snowballstemmer>=1.1 in c:\users\me\appdata\local\programs\python\python38\lib\site-packages (from sphinx) (2.0.0)
Requirement already satisfied, skipping upgrade: alabaster<0.8,>=0.7 in c:\users\me\appdata\local\programs\python\python38\lib\site-packages (from sphinx) (0.7.12)
Requirement already satisfied, skipping upgrade: setuptools in c:\users\me\appdata\local\programs\python\python38\lib\site-packages (from sphinx) (41.2.0)
Requirement already satisfied, skipping upgrade: colorama>=0.3.5; sys_platform == "win32" in c:\users\me\appdata\local\programs\python\python38\lib\site-packages (from sphinx) (0.4.4)
Requirement already satisfied, skipping upgrade: sphinxcontrib-jsmath in c:\users\me\appdata\local\programs\python\python38\lib\site-packages (from sphinx) (1.0.1)
Requirement already satisfied, skipping upgrade: babel>=1.3 in c:\users\me\appdata\local\programs\python\python38\lib\site-packages (from sphinx) (2.9.0)
Requirement already satisfied, skipping upgrade: imagesize in c:\users\me\appdata\local\programs\python\python38\lib\site-packages (from sphinx) (1.2.0)
Requirement already satisfied, skipping upgrade: sphinxcontrib-devhelp in c:\users\me\appdata\local\programs\python\python38\lib\site-packages (from sphinx) (1.0.2)
Requirement already satisfied, skipping upgrade: sphinxcontrib-qthelp in c:\users\me\appdata\local\programs\python\python38\lib\site-packages (from sphinx) (1.0.3)
Requirement already satisfied, skipping upgrade: Jinja2>=2.3 in c:\users\me\appdata\local\programs\python\python38\lib\site-packages (from sphinx) (2.11.2)
Requirement already satisfied, skipping upgrade: sphinxcontrib-applehelp in c:\users\me\appdata\local\programs\python\python38\lib\site-packages (from sphinx) (1.0.2)
Requirement already satisfied, skipping upgrade: requests>=2.5.0 in c:\users\me\appdata\local\programs\python\python38\lib\site-packages (from sphinx) (2.25.0)
Requirement already satisfied, skipping upgrade: packaging in c:\users\me\appdata\local\programs\python\python38\lib\site-packages (from sphinx) (20.4)
Requirement already satisfied, skipping upgrade: sphinxcontrib-htmlhelp in c:\users\me\appdata\local\programs\python\python38\lib\site-packages (from sphinx) (1.0.3)
Requirement already satisfied, skipping upgrade: Pygments>=2.0 in c:\users\me\appdata\local\programs\python\python38\lib\site-packages (from sphinx) (2.7.2)
Requirement already satisfied, skipping upgrade: pytz>=2015.7 in c:\users\me\appdata\local\programs\python\python38\lib\site-packages (from babel>=1.3->sphinx) (2020.4)
Requirement already satisfied, skipping upgrade: MarkupSafe>=0.23 in c:\users\me\appdata\local\programs\python\python38\lib\site-packages (from Jinja2>=2.3->sphinx) (1.1.1)
Requirement already satisfied, skipping upgrade: urllib3<1.27,>=1.21.1 in c:\users\me\appdata\local\programs\python\python38\lib\site-packages (from requests>=2.5.0->sphinx) (1.26.2)
Requirement already satisfied, skipping upgrade: idna<3,>=2.5 in c:\users\me\appdata\local\programs\python\python38\lib\site-packages (from requests>=2.5.0->sphinx) (2.10)
Requirement already satisfied, skipping upgrade: certifi>=2017.4.17 in c:\users\me\appdata\local\programs\python\python38\lib\site-packages (from requests>=2.5.0->sphinx) (2020.11.8)
Requirement already satisfied, skipping upgrade: chardet<4,>=3.0.2 in c:\users\me\appdata\local\programs\python\python38\lib\site-packages (from requests>=2.5.0->sphinx) (3.0.4)
Requirement already satisfied, skipping upgrade: six in c:\users\me\appdata\local\programs\python\python38\lib\site-packages (from packaging->sphinx) (1.15.0)
Requirement already satisfied, skipping upgrade: pyparsing>=2.0.2 in c:\users\me\appdata\local\programs\python\python38\lib\site-packages (from packaging->sphinx) (2.4.7)
Installing collected packages: sphinx
Successfully installed sphinx-3.3.1

But the folder c:\users\me\appdata\local\programs\python\python38\lib\site-packages\sphinx\ is still not there.

I even tried to run a new Sphinx project from scratch, using sphinx-quickstart:

For a list of supported codes, see
https://www.sphinx-doc.org/en/master/usage/configuration.html#confval-language.
> Project language [en]: en

Creating file C:\Users\me\Dropbox\Kuchentest\source\conf.py.
Creating file C:\Users\me\Dropbox\Kuchentest\source\index.rst.
Creating file C:\Users\me\Dropbox\Kuchentest\Makefile.
Creating file C:\Users\me\Dropbox\Kuchentest\make.bat.

Finished: An initial directory structure has been created.

You should now populate your master file C:\Users\me\Dropbox\Kuchentest\source\index.rst and create other documentation
source files. Use the Makefile to build the docs, like so:
   make builder
where "builder" is one of the supported builders, e.g. html, latex or linkcheck.

But despite this output, there are no such files or source folder being created.

What can I do to cleanly reset my Sphinx installation and get my documentation to run again?

Do not install packages into your system Python. It only leads to pain and suffering. Always create a virtual environment, then install packages into that. The process is (1) create a directory to hold your project, (2) `cd` into that directory, (3) `python -m venv env` to create the virtual environment, (4) `source env/bin/activate` to activate the virtual environment, (5) `env/bin/pip install Sphinx` to install Sphinx, (6) use Sphinx quick-start. — Steve Piercy, Nov 16 '20 at 04:36
@StevePiercy just to complement the above comment, I personally prefer to use my IDE's (PyCharm) interface when creating the venv. It's more intuitive, allows for browsing the PyPI package index by name, automates upgrading individual packages, etc...I have never taken the time to try using pip straight on the command line. — bad_coder, Nov 16 '20 at 11:59
@bad_coder because the OP however used command line, I responded in that context. I use PyCharm myself, but I configure it to disable `activate` (it's evil) in terminal sessions. Did you know that PyCharm usually does all those command line steps for you? It's good to know how to use them regardless of platform. — Steve Piercy, Nov 18 '20 at 04:34
@StevePiercy Thanks for your kind words. I understand now why you pointed out the command line steps. Yes I would assume PyCharm's UI runs pip cmds in the background for the user. What I didn't know is that precious detail about `activate` having nefarious effects in the editor's terminal - I'm sure to discover eventually and hopefully be reminded of your wise words on that day. — bad_coder, Nov 18 '20 at 04:51
FTR, I do `export VENV=/path/to/project/virtual/environment` then `$VENV/bin/pip` or whatever binary. Although a little more to type, it works from within any directory for my project. — Steve Piercy, Nov 18 '20 at 07:28

bad_coder · Answer 1 · 2020-11-16T02:54:12.437

Solving this requires a somewhat awkward explanation that is simultaneously dependent on: operating system (Windows), your particular installation, and how you are executing Sphinx.

On Windows you can have several Python installations in different places (depending...):

One usual location is C:\Program Files\Python3x.
The pre-configured default path is C:\Users\me\AppData\Local\Programs\Python\Python3.x\. I find this inconvenient because it's located deep away from the root.

The currently prevalent way of extending a Python installation is using a virtual environment (venv).

Your venv, wherever you decided to place it. (Using a venv is considered the "de facto" best practice.)

At one point in time, you set PYTHONPATH as an environment variable on Windows and it's in those paths that Windows will look for your Python installations. Notice the rules for Module Search Path. The problem now becomes that if you have more than one Python installation set on your Path, Windows will also look for libraries in other installations...

(A general note about Python installations on Windows is necessary. Sometime in 2019 Microsoft included Python with Windows - as noted by a prominent SO user in this answer, and referred in the documentation. Around that time there was a Windows bug that required environment variables to be set with the administrator account - I can't find a reference but it's mentioned somewhere on SO. Meaning it's advisable to make your separate installation of Python and set environment variables as admin.)

Having said that, the problem you are describing has several aspects (take special notice of the terminal you are using):

The first warning in your sphinx-build indicates Sphinx is trying to read files from your user account installation (point 2 above). The problem is the terminal where you are executing sphinx-build does not have permission to read from the user account installation directories, because the terminal is being run under a different user account or because the account installation paths aren't set with read permission... Having said that, reconsider the warnings:

copying static files... WARNING: Failed to copy a file in html_static_file: c:\users\me\appdata\local\programs\python\python38\lib\site-packages\sphinx\themes\basic\static/jquery-3.5.1.js: PermissionError(13, 'Permission denied')

WARNING: Failed to copy a file in html_static_file: c:\users\me\appdata\local\programs\python\python38\lib\site-packages\sphinx\themes\basic\static/jquery.js: PermissionError(13, 'Permission denied')

It may also be the case you deleted Sphinx from your account installation and the files/paths simply aren't there.

Next when you try to reinstall Sphinx using pip it is not entirely clear if it's an outdated cache issue, or if pip is finding Sphinx in another installation on your PYTHONPATH... It may be the case that Sphinx is installed and the terminal simply hasn't read/write permission (depends what user account invoked the terminal), or the directory may be hidden in the file explorer...

What can I do to cleanly reset my Sphinx installation and get my documentation to run again?

Your Python base installations (points 1 and 2 above) are only supposed to be written to for system or user wide changes (not for a particular project change).

It is strongly recommended that you use a venv. (If you haven't before this would be the right time to consider doing so because it's the easiest and cleanest solution). This may initially seem confusing because historically there have been several virtual-environments for Python. Currently venv is the most commonly cited solution and using it is simple, your IDE should have a built-in UI to help you create it with a couple of clicks.

A venv is a Python environment that extends your base installation, it avoids a need to change your base installation when you have to make project specific changes (like having Sphinx installed, should ideally be on the venv not the base installation.)

Finally, when you run Sphinx from the terminal it's advisable to activate your venv on the terminal, otherwise the Python installation that is executed may depend on the user account that invoked the terminal.

thank you for the detailed information, but to my understanding, my damaged package could also occur within a virtual environment, so I would need a remedy anyway? — Marie. P., Nov 16 '20 at 08:38
@Marie.P. Well saying 99% of users would discard the `venv` and make a new one is probably an understatement, because 100% of users would make a fresh `venv`. That's the whole point with keeping the base installation clean, nobody has to worry about how packages are fixed or managed. Your only fear is `pip` not working cleanly, in which case you also make a new `venv` and try again...`venv`s are *"discardable"* you don't worry about this sort of problem. — bad_coder, Nov 16 '20 at 11:37
Alright. I am looking into it. Now, I am stuck after creating a `venv` (https://docs.python.org/3/tutorial/venv.html, step 12.2). I ran `activate.bat`, but then `pip install sphinx` claims that everything is already installed, so the environment is seemingly not activated. — Marie. P., Nov 18 '20 at 20:22
@Marie.P. try `python -m pip sphinx`. Or, using the recursive upgrade option is generally recommended for Sphinx, also try [`pip -U sphinx`](https://pip.pypa.io/en/stable/user_guide/#only-if-needed-recursive-upgrade) with and without using `python -m` before the `pip` part. If it doesn't work, the problem will probably need some minute analyses [per these rules](https://pip.pypa.io/en/stable/user_guide/#user-installs). — bad_coder, Nov 18 '20 at 20:29
@Marie.P. also, if it doesn't work we need info about your exact spec. Please edit the question to include it: 1)Your windows version? 2)How did you set your environment variables, both with admin and user account? 3) Are you using an IDE, and which one? 4) Your terminal, are you running within the IDE or from the windows CMD or PowerShell? (This might be a corner case of using a rare combination of factors.) — bad_coder, Nov 18 '20 at 20:41

Removing and reinstalling Python-Sphinx on Windows

1 Answers1