pip install from cache is failing sometimes when no network

Question

I would like to install python packages offline. To do so, I collect them once online with pip install --cache-dir, then the next times, offline the same way, expecting pip will always read the cache.

The first time : online (empty cache, empty virtualenv):

~/tmp/venv$ LANG=C pip install --cache-dir /tmp/pipcache matplotlib
Collecting matplotlib
  Downloading matplotlib-2.0.2-cp27-cp27mu-manylinux1_x86_64.whl (14.5MB)
    100% |################################| 14.5MB 99kB/s 
Collecting numpy>=1.7.1 (from matplotlib)
  Downloading numpy-1.13.0-cp27-cp27mu-manylinux1_x86_64.whl (16.6MB)
    100% |################################| 16.6MB 86kB/s 
Collecting cycler>=0.10 (from matplotlib)
  Downloading cycler-0.10.0-py2.py3-none-any.whl
Collecting python-dateutil (from matplotlib)
  Downloading python_dateutil-2.6.0-py2.py3-none-any.whl (194kB)
    100% |################################| 194kB 5.8MB/s 
Collecting functools32 (from matplotlib)
  Downloading functools32-3.2.3-2.zip
Collecting six>=1.10 (from matplotlib)
  Downloading six-1.10.0-py2.py3-none-any.whl
Collecting pytz (from matplotlib)
  Downloading pytz-2017.2-py2.py3-none-any.whl (484kB)
    100% |################################| 491kB 3.1MB/s 
Collecting pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=1.5.6 (from matplotlib)
  Downloading pyparsing-2.2.0-py2.py3-none-any.whl (56kB)
    100% |################################| 61kB 8.8MB/s 
Collecting subprocess32 (from matplotlib)
  Downloading subprocess32-3.2.7.tar.gz (54kB)
    100% |################################| 61kB 8.4MB/s 
Building wheels for collected packages: functools32, subprocess32
  Running setup.py bdist_wheel for functools32 ... done
  Stored in directory: /tmp/pipcache/wheels/3c/d0/09/cd78d0ff4d6cfecfbd730782a7815a4571cd2cd4d2ed6e69d9
  Running setup.py bdist_wheel for subprocess32 ... done
  Stored in directory: /tmp/pipcache/wheels/7d/4c/a4/ce9ceb463dae01f4b95e670abd9afc8d65a45f38012f8030cc
Successfully built functools32 subprocess32
Installing collected packages: numpy, six, cycler, python-dateutil, functools32, pytz, pyparsing, subprocess32, matplotlib
Successfully installed cycler-0.10.0 functools32-3.2.3.post2 matplotlib-2.0.2 numpy-1.13.0 pyparsing-2.2.0 python-dateutil-2.6.0 pytz-2017.2 six-1.10.0 subprocess32-3.2.7

The Second time : offline (reusing cache, recreated empty virtualenv), everything is OK:

~/tmp/venv$ LANG=C pip install --cache-dir /tmp/pipcache matplotlib
Collecting matplotlib
  Using cached matplotlib-2.0.2-cp27-cp27mu-manylinux1_x86_64.whl
Collecting numpy>=1.7.1 (from matplotlib)
  Using cached numpy-1.13.0-cp27-cp27mu-manylinux1_x86_64.whl
Collecting cycler>=0.10 (from matplotlib)
  Using cached cycler-0.10.0-py2.py3-none-any.whl
Collecting python-dateutil (from matplotlib)
  Using cached python_dateutil-2.6.0-py2.py3-none-any.whl
Collecting functools32 (from matplotlib)
Collecting six>=1.10 (from matplotlib)
  Using cached six-1.10.0-py2.py3-none-any.whl
Collecting pytz (from matplotlib)
  Using cached pytz-2017.2-py2.py3-none-any.whl
Collecting pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=1.5.6 (from matplotlib)
  Using cached pyparsing-2.2.0-py2.py3-none-any.whl
Collecting subprocess32 (from matplotlib)
Installing collected packages: numpy, six, cycler, python-dateutil, functools32, pytz, pyparsing, subprocess32, matplotlib
Successfully installed cycler-0.10.0 functools32-3.2.3.post2 matplotlib-2.0.2 numpy-1.13.0 pyparsing-2.2.0 python-dateutil-2.6.0 pytz-2017.2 six-1.10.0 subprocess32-3.2.7

The third time : offline (reusing cache, recreated empty virtualenv), some errors:

~/tmp/venv$ LANG=C pip install --cache-dir /tmp/pipcache matplotlib
Collecting matplotlib
  Using cached matplotlib-2.0.2-cp27-cp27mu-manylinux1_x86_64.whl
Collecting numpy>=1.7.1 (from matplotlib)
  Using cached numpy-1.13.0-cp27-cp27mu-manylinux1_x86_64.whl
Collecting cycler>=0.10 (from matplotlib)
  Using cached cycler-0.10.0-py2.py3-none-any.whl
Collecting python-dateutil (from matplotlib)
  Using cached python_dateutil-2.6.0-py2.py3-none-any.whl
Collecting functools32 (from matplotlib)
Collecting six>=1.10 (from matplotlib)
  Retrying (Retry(total=4, connect=None, read=None, redirect=None)) after connection broken by 'NewConnectionError('<pip._vendor.requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f7abfbdbfd0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',)': /simple/six/
  Retrying (Retry(total=3, connect=None, read=None, redirect=None)) after connection broken by 'NewConnectionError('<pip._vendor.requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f7abfbdb490>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',)': /simple/six/
  Retrying (Retry(total=2, connect=None, read=None, redirect=None)) after connection broken by 'NewConnectionError('<pip._vendor.requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f7abfbdbf50>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',)': /simple/six/
  Retrying (Retry(total=1, connect=None, read=None, redirect=None)) after connection broken by 'NewConnectionError('<pip._vendor.requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f7ac1068ed0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',)': /simple/six/
  Retrying (Retry(total=0, connect=None, read=None, redirect=None)) after connection broken by 'NewConnectionError('<pip._vendor.requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f7ac1068910>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',)': /simple/six/
  Could not find a version that satisfies the requirement six>=1.10 (from matplotlib) (from versions: )
No matching distribution found for six>=1.10 (from matplotlib)

There are network errors : I do not know why because all packages are available in the cache. In addition, the errors are random : it could occur when installing six or pytz package. Sometimes there may be no error at all while sometimes the above happens.

How to do I correct this, so I can do an offline installation?

Note : pip and setuptools are at their latest versions

According to https://stackoverflow.com/questions/4806448/how-do-i-install-from-a-local-cache-with-pip, you don't need the --cache-dir option since caching is on by default. — Liam Bohl, Jun 27 '17 at 10:18

AKX · Accepted Answer · 2017-06-27T10:32:54.847

Try adding the --no-index parameter (and possibly --find-links too).

EDIT: Using pip download and pip install -f seems to work:

$ docker run -it python:3.6 bash

root@06dbe462c2c3:/# cd tmp
root@06dbe462c2c3:/tmp# ls
root@06dbe462c2c3:/tmp# mkdir cache
root@06dbe462c2c3:/tmp# pip download -d cache matplotlib
Successfully downloaded matplotlib pytz numpy six pyparsing cycler python-dateutil
root@06dbe462c2c3:/tmp# pip install --no-index matplotlib
No matching distribution found for matplotlib
root@06dbe462c2c3:/tmp# pip install --no-index matplotlib -f cache/
Collecting matplotlib
Collecting cycler>=0.10 (from matplotlib)
Collecting pyparsing!=2.0.0,!=2.0.4,!=2.1.2,!=2.1.6,>=1.5.6 (from matplotlib)
Collecting python-dateutil (from matplotlib)
Collecting pytz (from matplotlib)
Collecting numpy>=1.7.1 (from matplotlib)
Collecting six>=1.10 (from matplotlib)
Installing collected packages: six, cycler, pyparsing, python-dateutil, pytz, numpy, matplotlib
Successfully installed cycler-0.10.0 matplotlib-2.0.2 numpy-1.13.0 pyparsing-2.2.0 python-dateutil-2.6.0 pytz-2017.2 six-1.10.0
root@06dbe462c2c3:/tmp#

I tried many combinaisons, but with --no-index I have "Could not find a version that satisfies the requirement matplotlib", with --find-links pointing to cache I have the same behaviour : random errors. — Eric, Jun 27 '17 at 10:19
I tested without network : it worked fine, even with a big requirement file. thank you very much ! — Eric, Jun 27 '17 at 12:24

pip install from cache is failing sometimes when no network

1 Answers1