Python setuptools: How do I specify dependencies in a private PyPI repo, hosted on Nexus?

Question

I'm unable to build a python wheel which has a dependency on a package in a private PyPI repository (hosted on Sonatype Nexus Repository Manager, vOSS 3.17.0-01).

I can search for and install the package if I use pip; my problem is trying to get setup.py to do the same. From looking into the output of various commands, I think the problem may be due to relative paths in the package links provided by the repository.

Searching for similar issues, most of them concern private git repositories. The most seemingly relevant related issue I could find is Equivalent for `--find-links` in `setup.py`. The body of that comment states that

In a setuptools context the dependency_links option accepts ... the URLs of web pages that contain direct download links

However, the linked page is offered in support of that quote no longer contains that text (replaced by a newer version?)

So that may be the problem - and the way I'm trying to do things is not supported. If that's the case, can someone suggest a way that works?

Alternatively, this could be a bug in setuptools, or a misconfiguration problem with our Nexus - and if anyone could confirm or refute those theories - and again, suggest an approach that works - I'd appreciate it.

Here's my setup and the output of various commands, showing what works and what doesn't:

Set up the environment:

mkdir depends-fail
cd depends-fail
python3 -m venv venv
source activate venv/bin/activate
pip install --upgrade pip
pip install wheel        # So we can use bdist_wheel build option.

Confirm our current pip version:

(venv) $ pip --version
pip 21.0.1 from /tmp/depends-fail/venv/lib/python3.6/site-packages/pip (python 3.6)

Confirm setuptools version:

(venv) $ python -c "import setuptools as s; print(s.version.__version__)"
39.0.1

In the depends-fail directory, create this minimal setup.py file, which will demonstrate the problem:

 from setuptools import setup
 setup(
   name='failed-dependencies',
   install_requires=['data-feed-ping>=0.5'],
   dependency_links=[
     'http://nexus.example.local/nexus/repository/pypi-playground-public/simple/data-feed-ping',
   ]
)

Confirm that our dependency is available in our private repository:

 (venv) $ pip search --trusted-host nexus.example.local \
 >  -i http://nexus.example.local/nexus/repository/pypi-playground-public/pypi \
 >  data-feed-ping

The response confirms the package is locally available:

⋮
Starting new HTTP connection (1): nexus.example.local:80
http://nexus.example.local:80 "POST /nexus/repository/pypi-playground-public/pypi HTTP/1.1" 200 239
data-feed-ping (0.5)  - Determine response times of data feeds.

Here's what happens when we try to build and test our minimal package (it's the test option which triggers the dependency downloads):

(venv) $ python setup.py bdist_wheel test

The package builds without problem - but the dependencies can't be found:

    running bdist_wheel
    running build
    ⋮
    removing build/bdist.linux-x86_64/wheel
    running test
    Searching for data-feed-ping>=0.5
(1) Reading http://nexus.example.local/nexus/repository/pypi-playground-public/simple/data-feed-ping
(2) Downloading http://nexus.example.local/nexus/repository/packages/data-feed-ping/0.5/data_feed_ping-0.5-py3-none-any.whl#md5=e7a9ee0be6cc77165d02e7022c04b336
    error: Can't download http://nexus.example.local/nexus/repository/packages/data-feed-ping/0.5/data_feed_ping-0.5-py3-none-any.whl#md5=e7a9ee0be6cc77165d02e7022c04b336: \
    404 Repository not found

The download link and version hash in (2) come from the contents of the index URL read at (1) - so let's see what that file looks like:

(venv) $ curl http://nexus.example.local/nexus/repository/pypi-playground-public/simple/data-feed-ping

<html lang="en">
<head><title>Links for data-feed-ping</title><meta name="api-version" value="2"/></head>
  <body><h1>Links for data-feed-ping</h1>
    ⋮  <!-- links to previous versions -->
    <a href="../../packages/data-feed-ping/0.5/data_feed_ping-0.5-py3-none-any.whl#md5=e7a9ee0be6cc77165d02e7022c04b336" rel="internal">data_feed_ping-0.5-py3-none-any.whl</a><br/>
  </body>
</html>

The index URL does provide a link to the required version of the data-feed-ping package, and the setup script is correctly picking up the md5 digest from that link (see (2) in the script output). However, the setup script then tries to download the file from an invalid URL.

To me it looks as if the problem is the relative path provided by the index link. If we start with the index URL (1):

http://nexus.example.local/nexus/repository/pypi-playground-public/simple/data-feed-ping

and add the relative path from the download:

../../packages/data-feed-ping/0.5/data_feed_ping-0.5-py3-none-any.whl#md5=e7a9ee0be6cc77165d02e7022c04b336

I get the following absolute path:

http://nexus.example.local/nexus/repository/pypi-playground-public/packages/data-feed-ping/0.5/data_feed_ping-0.5-py3-none-any.whl#md5=e7a9ee0be6cc77165d02e7022c04b336

I've confirmed that that absolute path leads to the package I want:

(venv) $ wget http://nexus.example.local/nexus/repository/pypi-playground-public/packages/data-feed-ping/0.5/data_feed_ping-0.5-py3-none-any.whl#md5=e7a9ee0be6cc77165d02e7022c04b336

--2021-04-01 16:44:18--  http://nexus.example.local/nexus/repository/pypi-playground-public/packages/data-feed-ping/0.5/data_feed_ping-0.5-py3-none-any.whl
Resolving nexus.example.local (nexus.example.local)... 192.168.24.136
Connecting to nexus.example.local (nexus.example.local)|192.168.24.136|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 19066 (19K) [application/zip]
Saving to: ‘data_feed_ping-0.5-py3-none-any.whl’
data_feed_ping-0.5-py3-none-any 100%[==========================================>]  18.62K  --.-KB/s    in 0s
2021-04-01 16:44:18 (76.3 MB/s) - ‘data_feed_ping-0.5-py3-none-any.whl’ saved [19066/19066]

I also have no problem if I try and download the package directly, with pip, specifying the same index URL that the setup script uses:

   (venv) $ pip install --trusted-host nexus.example.local \
   >  --index-url http://nexus.example.local/nexus/repository/pypi-playground-public/simple \
   >  data-feed-ping

   Looking in indexes: http://nexus.example.local/nexus/repository/pypi-playground-public/simple
Collecting data-feed-ping
   Downloading http://nexus.example.local/nexus/repository/pypi-playground-public/packages/data-feed-ping/0.5/data_feed_ping-0.5-py3-none-any.whl (19 kB)
   Requirement already satisfied: requests in ./venv/lib/python3.6/site-packages (from data-feed-ping) (2.25.1)
   ⋮
   Installing collected packages: data-feed-ping
   Successfully installed data-feed-ping-0.5

You should only use `pip` (or any other installer) to install Python projects. Installing by calling `python setup.py install` for example is a bad practice. So in consequence, you probably should remove the `dependency_links` parameter of your `setup.py`, and instruct your users to install with the correct _pip_ flags exactly as you have shown at the end of your question. — sinoroc, Apr 01 '21 at 17:44

sinoroc · Accepted Answer · 2022-03-28T18:38:19.697

0

You should only use pip (or any other installer) to install Python projects. Installing by calling python setup.py install for example is an outdated practice (that often does not work). So in consequence, you probably should remove the dependency_links parameter of your setup.py, and instruct your users to install with the correct pip flags exactly as you have shown at the end of your question:

python -m pip install --trusted-host nexus.example.local \
   >  --index-url http://nexus.example.local/nexus/repository/pypi-playground-public/simple \
   >  data-feed-ping

In the same way python setup.py test: is outdated/deprecated as well, and you might be well-advised to move on to a more modern test runner (pytest is the one that comes to mind, but there are probably others, maybe nose).

References:

edited Mar 28 '22 at 18:38

answered Apr 01 '21 at 17:44

sinoroc

18,409
2
39
70

Sorry, but under the hood doesn't running `pip install somepackage` mean running the `setup.py` file for that package? Isn't that what python would *have* to do? – Jordan Reiter Mar 28 '22 at 16:18
@jordanreiter First, not all libraries have a `setup.py` file. If there is a `setup.py`, chances are that _pip_ will run `setup.py` at some point (at least once to build a wheel, and then pip will use the wheel for subsequent installations). But I am honestly not sure that pip will run `setup.py install` at all, I think probably pip will run other commands of `setup.py`. -- I am not sure what your question really is about. Anyway, lots of details are available to read here: https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html – sinoroc Mar 28 '22 at 18:37

Python setuptools: How do I specify dependencies in a private PyPI repo, hosted on Nexus?

1 Answers1

Linked