7

I have recently found myself in the situation of having to install all dependencies (20+) of a python project on a machine without internet connection. I used pip download ... to get all the *.whl files and transferred them manually. Only now I fully appreciate the genius of pip and how it figures out the dependency tree on its own and manages to install every package in the right order. E.g. a package depends on the requests package which on its own depends on the urllib3 package and so on.

I wanted an automated way to install all these dependencies on the machine using the command console or python itself, so I turned to StackOverflow and found these solutions: How to install multiple whl files in cmd

Nearly all suggested solutions work for me, however have the disadvantage of having to run them multiple times until no installation fails anymore! This is due to the scripts/commands sorting the packages alphabetically and trying to install them in that order (e.g. trying to install requests before urllib3 is in place).

Is there a smarter way to do this with only executing a script/command on time?

sinoroc
  • 18,409
  • 2
  • 39
  • 70
Sebastian Dengler
  • 1,258
  • 13
  • 30
  • 1
    As a frame challenge - have you looked at "hosting" your own PyPI repository on the machine without an internet connection? https://stackoverflow.com/a/51127695/1149933 – Adam Barnes Jan 22 '21 at 12:03
  • No, until this moment I did not know that was possible. Do you have recommendations/links for material explaining the process? – Sebastian Dengler Jan 22 '21 at 12:05
  • 1
    I updated my comment with a link to an answer to just that question (I think) while you were typing that response. – Adam Barnes Jan 22 '21 at 12:06
  • 1
    I am a bit confused. Why does the order matter? It definitely should not matter. If it does matter, then something might be broken somewhere... Installing _wheels_ is only a matter of unzipping the wheels files into the right folders (a bit more complicated, but not that much). Have you tried adding a `--no-deps` in your installation command? Maybe something like `python -m pip install --no-deps wheels/*.whl`? – sinoroc Jan 23 '21 at 11:13
  • This is it! Since I'm sure that I have all dependencies in form of .whl files it is justifiable to use the `--no-deps` flag. Although Adam's suggestion works as well this is of course simpler. If you want to make it an answer I will accept it. – Sebastian Dengler Jan 23 '21 at 16:15

2 Answers2

9

This directory full of wheel distribution files that you created is sometimes called a wheelhouse. They are often used to make repeatable and/or offline installations.

A common way to install from a wheelhouse is:

python -m pip install --no-index --no-deps path/to/wheelhouse/*.whl

If all the wheels for all dependencies and their dependencies are in the wheelhouse, then the installation order does not matter and there is no need to connect to any remote package index (for dependency resolution, etc.). That is why we can use the --no-deps and --no-index flags.

Reference:

Aside:

Only now I fully appreciate the genius of pip and how it figures out the dependency tree on its own and manages to install every package in the right order.

pip's dependency resolver is resolvelib. There is a simple example on its source code repository showing how to use it to solve for wheels on PyPI.

sinoroc
  • 18,409
  • 2
  • 39
  • 70
1

Easiest way of figuring the correct installation order is to sort packages according to their dependencies. Those topics were already discussed separately in couple other SO questions, so I will link to them instead of repeating. The "Safe installation algorithm" looks as follows:

  1. Read metadata of all packages. this question explains how Focus on package name and dependencies.
  2. Sort them topologically Article on Wikipedia and another helpful question
  3. Install packages according to the order.
majkrzak
  • 1,332
  • 3
  • 14
  • 30
  • Although this is a very systematic way of tackling the problem I feel like this answer is missing the point of the question. The aim was to find a way of automating the process in one script or command. Before trying to find out the dependency graph manually and digging through the metadata of the packages I can also simply execute my script n times where n in the worst case is the depth of the graph until no installation fails anymore. – Sebastian Dengler Jan 23 '21 at 16:19
  • For info: _pip_'s dependency resolver is [_resolvelib_](https://pypi.org/project/resolvelib/). There is also a [simple example](https://github.com/sarugaku/resolvelib/blob/e91ef74ad0aec4795e2ffc1ef1686004c3dbd6be/examples/pypi_wheel_provider.py) showing how to use it to solve for _wheels_ on _PyPI_. – sinoroc Jan 23 '21 at 18:31