How can I figure out exactly which package is causing pip backtracking issues?

Question

I've got a repo with a set of different requirements.txt for a few different cloud functions. I separate the install/test for each with a loop in my bitbucket-pipelines.yml:

for d in `find . -type d -maxdepth 1 -mindepth 1`; do
  if [ -f "$d/requirements.txt" ]; then
    echo "====="$d"====="
    python3 -m pip install -r $d/requirements.txt
    python3 -m coverage run -a --omit="*test*" -m unittest discover -v $d
  fi
done

It works great except for one of my functions, where pip spins its wheels (pun intended). This function happens to depend on Tensorflow, which has all kinds of deeper Google dependencies, and it seems like these don't get along with other function dependencies. But you may not know this from pip, because it just produces a series of messages like

Collecting pandas
  Downloading pandas-1.5.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 12.2/12.2 MB 46.0 MB/s eta 0:00:00
  Downloading pandas-1.5.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 12.2/12.2 MB 41.6 MB/s eta 0:00:00
  Downloading pandas-1.4.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.7/11.7 MB 62.0 MB/s eta 0:00:00
Requirement already satisfied: charset-normalizer<3,>=2 in /usr/local/lib/python3.9/site-packages (from requests->-r ./sleep/requirements.txt (line 5)) (2.1.1)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.9/site-packages (from requests->-r ./sleep/requirements.txt (line 5)) (2022.9.24)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.9/site-packages (from requests->-r ./sleep/requirements.txt (line 5)) (1.26.13)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.9/site-packages (from requests->-r ./sleep/requirements.txt (line 5)) (3.4)
Requirement already satisfied: setuptools in /usr/local/lib/python3.9/site-packages (from grpcio-tools>=1.44.0->-r ./sleep/requirements.txt (line 7)) (58.1.0)
Collecting data-consumption-interface>=1.11
  Downloading https://pypi.biointellisense.com/packages/data-consumption-interface/1.11.0/data_consumption_interface-1.11.0-py3-none-any.whl (53 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 53.6/53.6 kB 65.1 MB/s eta 0:00:00
INFO: pip is looking at multiple versions of grpcio-tools to determine which version is compatible with other requirements. This could take a while.
Collecting grpcio-tools>=1.44.0
  Downloading grpcio_tools-1.50.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.4 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.4/2.4 MB 68.6 MB/s eta 0:00:00
INFO: pip is looking at multiple versions of requests to determine which version is compatible with other requirements. This could take a while.
Collecting requests
  Downloading https://pypi.biointellisense.com/packages/requests/2.28.1/requests-2.28.1-py3-none-any.whl (62 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.8/62.8 kB 70.2 MB/s eta 0:00:00
INFO: pip is looking at multiple versions of pandas to determine which version is compatible with other requirements. This could take a while.
Collecting pandas
  Downloading pandas-1.4.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.7/11.7 MB 100.7 MB/s eta 0:00:00
  Downloading pandas-1.4.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.7/11.7 MB 94.2 MB/s eta 0:00:00
  Downloading pandas-1.4.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.7/11.7 MB 90.2 MB/s eta 0:00:00
  Downloading pandas-1.4.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.7/11.7 MB 92.5 MB/s eta 0:00:00
  Downloading pandas-1.3.5-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.5/11.5 MB 106.1 MB/s eta 0:00:00
  Downloading pandas-1.3.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.5/11.5 MB 118.8 MB/s eta 0:00:00
  Downloading pandas-1.3.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.5/11.5 MB 102.0 MB/s eta 0:00:00
INFO: pip is looking at multiple versions of pandas to determine which version is compatible with other requirements. This could take a while.
  Downloading pandas-1.3.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.5/11.5 MB 83.4 MB/s eta 0:00:00
  Downloading pandas-1.3.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.7/11.7 MB 107.7 MB/s eta 0:00:00
  Downloading pandas-1.3.0-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.whl (10.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.6/10.6 MB 122.8 MB/s eta 0:00:00
  Downloading pandas-1.2.5-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.whl (9.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 9.7/9.7 MB 111.9 MB/s eta 0:00:00
  Downloading pandas-1.2.4-cp39-cp39-manylinux1_x86_64.whl (9.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 9.7/9.7 MB 100.2 MB/s eta 0:00:00
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See https://pip.pypa.io/warnings/backtracking for guidance. If you want to abort this run, press Ctrl + C.
  Downloading pandas-1.2.3-cp39-cp39-manylinux1_x86_64.whl (9.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 9.7/9.7 MB 116.0 MB/s eta 0:00:00
  Downloading pandas-1.2.2-cp39-cp39-manylinux1_x86_64.whl (9.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 9.7/9.7 MB 113.7 MB/s eta 0:00:00
  Downloading pandas-1.2.1-cp39-cp39-manylinux1_x86_64.whl (9.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 9.7/9.7 MB 106.8 MB/s eta 0:00:00
  Downloading pandas-1.2.0-cp39-cp39-manylinux1_x86_64.whl (9.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 9.7/9.7 MB 110.1 MB/s eta 0:00:00
  Downloading pandas-1.1.5-cp39-cp39-manylinux1_x86_64.whl (9.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 9.3/9.3 MB 84.5 MB/s eta 0:00:00
  Downloading pandas-1.1.4-cp39-cp39-manylinux1_x86_64.whl (9.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 9.3/9.3 MB 110.9 MB/s eta 0:00:00
  Downloading pandas-1.1.3-cp39-cp39-manylinux1_x86_64.whl (9.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 9.3/9.3 MB 120.0 MB/s eta 0:00:00
  Downloading pandas-1.1.2.tar.gz (5.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.2/5.2 MB 94.3 MB/s eta 0:00:00
  Installing build dependencies: started
  Installing build dependencies: still running...
  Installing build dependencies: still running...
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: still running...
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
  Downloading pandas-1.1.1.tar.gz (5.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.2/5.2 MB 71.5 MB/s eta 0:00:00
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started

...on and on.

I can try to set specific versions of certain libraries in that function's requirements.txt, but then I get this kind of infinite regress for other libraries, like tensorflow, urllib, or cffi.

It truly does not make sense that pip should have to download and try to install entire packages before it determines the versions aren't compatible. That should all be available in metadata. Why does python work this way? https://pip.pypa.io/en/stable/topics/dependency-resolution/

I can't tell what specifically is causing this problem. How can I determine from the logs exactly which package I need to == version? Pip is never giving me an error message about what is interfering to cause it to try to roll versions back! The community needs to fix this.

This is fragile. I've been able to reconcile dependencies a few times by playing with it long enough, but that approach can't scale. And then when some new version of some deep dependency is published, pip loses all track and has to do complicated resolution all over again.

If you want deterministic behavior, I'd switch to using Nix for dependency management. Short of that, though -- can you build a [MRE] others can run to see this issue? — Charles Duffy, Nov 29 '22 at 21:55
I *would* separate them when testing in your pipeline, and this is precisely why. I haven't used bitbucket's version, but most other providers allow you some form of parallelism for tasks. If they are going to run in separate environments, they should be built and tested separately as well — C.Nivs, Nov 29 '22 at 21:56
@Pavel Komarov - thank you for bringing this up. It's funny that this question has only 112 views as of 2023-01-29 and there is not more activity on how annoying this all is. IMHO it's a bad idea to try the magic things pip tries. I'd rather see a certain dependency be installed and my tests will catch the problem. Then i am usually in the positition to find the exact patch/version change needed to fix the problem. Trying the general approach is IMHO bound for too much complexity and a direct way into general failure. — Wolfgang Fahl, Jan 29 '23 at 18:44

score -3 · Answer 1 · answered Nov 29 '22 at 22:54

As it seems your question has been answered in another Question

A bit of an explanation:

First we assume we work with python-version-2.5 (because I chose so - you have to replace 2.5 by your python-version)

First there was/is the command to avoid multiple requests: ex.g. $ python-2.5 -m pip install myfoopackage #you avoid them by specifying your python version (*as it didn't worked in your case I assume you put the wrong python version which you can find out with $ python -V)
Since version 0.8, Pip supports pip-{python-version} ex.g. pip-2.5 install myfoopackage #as you can see python-<VERSION> -m is no longer needed
Since version 1.5 they changed the schema to pip<VERSION> ex.g. pip2.5 intall mypackage

TL;DR

TRY:

$ python -V
$ for f in find . -type f -iname "requirements*.txt"; do
  pip<python_versoin> install -r $f
done

@PavelKomarov which version exactly? – Claude Rollinger Nov 29 '22 at 23:11 — Claude Rollinger, Nov 29 '22 at 23:11

How can I figure out exactly which package is causing pip backtracking issues?

1 Answers1