pip3 command to upgrade all packages, that is careful about dependency conflicts?

Question

So far, I have used (via How to upgrade all Python packages with pip)

pip3 list --format freeze --outdated | cut -d= -f1 | xargs pip3 install --upgrade-strategy eager --upgrade

to upgrade all of my Python pip packages. It has so far worked fine for me - except for once, when I got a sort of a conflict message, unfortunately I didn't keep a copy of it; my guess is, it was something similar to this noted here https://pip.pypa.io/en/stable/user_guide/#fixing-conflicting-dependencies :

Due to conflicting dependencies pip cannot install
package_coffee and package_tea:
- package_coffee depends on package_water<3.0.0,>=2.4.2
- package_tea depends on package_water==2.3.1

Anyways, now I just tried to install voila for my Jupyter installation, and it ended up like this:

(notebook) user@server:/home/web/Jupyter$ pip3 install voila
...
Installing collected packages: jupyter-client, voila
  Attempting uninstall: jupyter-client
    Found existing installation: jupyter-client 7.0.3
    Uninstalling jupyter-client-7.0.3:
      Successfully uninstalled jupyter-client-7.0.3
Successfully installed jupyter-client-6.1.12 voila-0.2.13

In other words: I've had jupyter-client-7.0.3 installed before as latest; but now that I wanted to install voila, due to voila requirements, that latest version got uninstalled, and an earlier version, 6.1.12, compatible with voila, got installed instead.

So now if I want to check outdated packages, I get, as expected, jupyter-client listed:

(notebook) user@server:/home/web/Jupyter$ pip3 list --format freeze --outdated
jupyter-client==6.1.12

... but then, if I run the full pipe command, pip3 list --format freeze --outdated | cut -d= -f1 | xargs pip3 install --upgrade-strategy eager --upgrade, then it will want to upgrade jupyter-client to 7.0.3, which will then break voila (I guess, I dare not try it).

So, is there an upgrade command, that would take a situation like this, and upon such a state during upgrade, prevent changes and give me a notification? Say, something like:

WARNING: There is an upgrade to jupyter-client=6.1.12 (newest version 7.0.3) - however, installing that package would cause a conflict with the currently installed voila=0.2.13 package; not proceeding with this upgrade. To force this upgrade regardless, use [...]

Answer 1

Upgrading packages in python is never easy due to overlapping (sub)dependencies. There are some tools out there that try and help you manage. At my current job we use pip-tools. And in some projects we use poetry but I'm less happy about it's handling.

For pip-tools you define your top-level packages in requirements.in file, which then resolves the sub(sub-sub)dependencies and outputs them into a requirements.txt file. The benefit of this is that you only worry about your main packages. You can still upgrade sub dependencies if so desired.

Long story short; blindly updating all your packages will most likely never work out as intended or expected. Either packages ARE upgraded, but stop working, or they do work but don't work with another package that was updated because they needed a lower version of that package.

My advice would be to start with your main packages and build up from there using one of the tools mentioned. There isn't a silver bullet for this. Dependency hell is a very real thing in python.