How to specify recursive dependencies in setup.py?

Question

I’m having difficulties understanding and implementing a few details of Python packaging using pip and setuptools.

Suppose I have three projects, A and B and C each of which is its own package and hosted in its own code repo. They also depend on each other, i.e. A is imported by B, and B is imported by C. Each package has a set of direct dependencies (i.e. directly imported other packages) and a set of indirect dependencies (i.e. packages imported by the directly dependent packages). These dependencies are a graph, not a tree.

For package A, should setup.py contain only the directly dependent packages? Same for package B? When I then pip install C I noticed that B gets installed, but not A. I suppose that is because A is an indirect dependency for C.

I don’t really like the idea of storing a pip freeze in each package (inflexible and conflicts loom), but it seems that pip does not resolve the dependency graph recursively (see here). The snakebasket project attempted to address that problem, but is stale now; the dependency-links option has been deprecated.

What is the proper and recommended way of handling this?

Addendum I forgot to mention that none of the packages (A, B, C) are available through the official PyPi repo, but live in private Github repos. Thus, for example B’s setup.py contains

install_requires=(
    …,
    A==1.0.0,
    …,
    )

dependency_links=[                                                                                        
    f"https://{github_token}@github.com/repo/A/archive/v1.0.0.tar.gz#egg=A-1.0.0",
    ],

and C contains a similar setup for package B.

score 0 · Answer 1 · answered Apr 26 '19 at 18:36

For package A, should setup.py contain only the directly dependent packages?

Yes. Separation of concerns: every package should list dependencies it requires. Subpackages should care for themselves.

When I then pip install C I noticed that B gets installed, but not A.

Can you show an example? I have a different experience.

it seems that pip does not resolve the dependency graph recursively (see here).

The answer from 2015 is outdated. pip install and pip download installs and downloads dependencies recursively.

I apologize, I should have mentioned that the packages live in a private repo. See also the **Addendum** above — Jens, Apr 27 '19 at 03:47

wim · Accepted Answer · 2019-04-26T20:04:46.500

0

How to specify recursive dependencies in setup.py?

Don't.

What is the proper and recommended way of handling this?

setup.py should list the direct dependencies under install_requires=[...]. Do not list the transitive dependencies. And do not pin the dependencies here (though, in some cases, you may want to specify an upper or lower bound to ensure a compatible version is collected).

When I then pip install C I noticed that B gets installed, but not A.

Then B is not correctly specifying the dependency on A. Check again the metadata of B.

It is true that pip sometimes fails to resolve the dependency tree correctly, and there is an open issue about that since 2013, but you won't see it in a simple C -> B -> A dependency graph, only in some more pathological cases.

Check out my project johnnydep to render the dep tree and indicate where you're missing a "branch" in the package metadata.

edited Apr 26 '19 at 20:04

answered Apr 26 '19 at 18:44

wim

338,267
99
616
750

I apologize, I should have mentioned that the packages live in a private repo. See also the **Addendum** above – Jens Apr 27 '19 at 03:47
1

Yes, that was critical info to include! `dependency_links` has been deprecated for some time. The best approach is to host packages to a package index - not necessarily PyPI, it can be a private index - rather than trying to depend direct from github (which does not satisfy the [Simple Repository API](https://www.python.org/dev/peps/pep-0503/) and so will not work well with `pip`). – wim Apr 27 '19 at 03:56
Thanks! Can you recommend such a private index or will I have to host one myself? Is [this article](https://packaging.python.org/guides/hosting-your-own-index/) still valid? – Jens Apr 27 '19 at 04:08
Yes that article is still valid. I recommend using a [devpi-server](https://github.com/devpi/devpi). – wim Apr 27 '19 at 04:27
One more question: is it possible to specify a private index in setup.py? I see that `pip -i` would work, or adding `index-url` to `~/.pip/pip.conf` ([here](https://stackoverflow.com/questions/22052288/install-python-package-from-private-pypiserver#22053466)). However, I think it would make much sense to specify my private index server as part of setup.py (I think `dependency_links` handled that). Also, what do you think of [this approach](https://medium.freecodecamp.org/how-to-use-github-as-a-pypi-server-1c3b0d07db2)? – Jens May 02 '19 at 00:15
1

No, it is not possible. The *user* has to opt-in to using a non-standard index, not the package. It will be too easy for hackers to distribute malicious code if you could put in the `setup.py` metadata to get a dependency from some random website instead of from PyPI. – wim May 02 '19 at 01:43
As for the article, no, I do not think much of that approach using github.com as a package index. It looks hacky and lame. There are perfectly good and free open source index servers available for this, no need to abuse a version control system for the same (it's like using a screwdriver to hammer a nail: github is for managing source code, not distributing packages). Further, an index should ideally support search, upload, json api, metadata, etc.. and user would still need to opt-in to the --extra-index-url anyway, so it doesn't really help in your setup.py either. – wim May 02 '19 at 01:52

How to specify recursive dependencies in setup.py?

2 Answers2