2

This is an extension of SO setup.py ignores full path dependencies, instead looks for "best match" in pypi

I am trying to write setup.py to install a proprietary package from a .tar.gz file on an internal web site. Unfortunately for me the prop package name duplicates a public package in the public PyPI, so I need to force install of the proprietary package at a specific version. I'm building a docker image from a Debian-Buster base image, so pip, setuptools and tox are all freshly installed, the image brings python 3.8 and pip upgrades itself to version 21.2.4.

Solution 1 - dependency_links

I followed the instructions at the post linked above to put the prop package in install_requires and dependency_links. Here are the relevant lines from my setup.py:

    install_requires=["requests", "proppkg==70.1.0"],
    dependency_links=["https://site.mycompany.com/path/to/proppkg-70.1.0.tar.gz#egg=proppkg-70.1.0"]

Installation is successful in Debian-Buster if I run python3 setup.py install in my package directory. I see the proprietary package get downloaded and installed.

Installation fails if I run pip3 install . also tox (version 3.24.4) fails similarly. In both cases, pip shows a message "Looking in indexes" then fails with "ERROR: Could not find a version that satisfies the requirement".

Solution 2 - PEP 508

Studying SO answer pip ignores dependency_links in setup.py which states that dependency_links is deprecated, I started over, revised setup.py to have:

    install_requires=[
        "requests", 
        "proppkg @ https://site.mycompany.com/path/to/proppkg-70.1.0.tar.gz#egg=proppkg-70.1.0"
    ],

Installation is successful in Debian-Buster if I run pip3 install . in my package directory. Pip shows a message "Looking in indexes" but still downloads and installs the proprietary package successfully.

Installation fails in Debian-Buster if I run python3 setup.py install in my package directory. I see these messages:

Searching for proppkg@ https://site.mycompany.com/path/to/proppkg-70.1.0.tar.gz#egg=proppkg-70.1.0
..
Reading https://pypi.org/simple/proppkg/
..
error: Could not find suitable distribution for Requirement.parse(...).

Tox also fails in this scenario as it installs dependencies.

Really speculating now, it almost seems like there's an ordering issue. Tox invokes pip like this:

python -m pip install --exists-action w .tox/.tmp/package/1/te-0.3.5.zip

In that output I see "Collecting proppkg@ https://site.mycompany.com/path/to/proppkg-70.1.0.tar.gz#egg=proppkg-70.1.0" as the first step. That install fails because it fails to import package requests. Then tox continues collecting other dependencies. Finally tox reports as its last step "Collecting requests" (and that succeeds). Do I have to worry about ordering of install steps?

I'm starting to think that maybe the proprietary package is broken. I verified that the prop package setup.py has requests in its install_requires entry. Not sure what else to check.

Workaround solution

My workaround is installing the proprietary package in the docker image as a separate step before I install my own package, just by running pip3 install https://site.mycompany.com/path/to/proppkg-70.1.0.tar.gz. The setup.py has the PEP508 URL in install_requires. Then pip and tox find the prop package in the pip cache, and work fine.

Please suggest what to try for the latest pip and tox, or if this is as good as it gets, thanks in advance.

Update - add setup.py

Here's a (slightly sanitized) version of my package's setup.py

from setuptools import setup, find_packages


def get_version():
    """
    read version string
    """
    version_globals = {}
    with open("te/version.py") as fp:
        exec(fp.read(), version_globals)
    return version_globals['__version__']


setup(
    name="te",
    version=get_version(),
    packages=find_packages(exclude=["tests.*", "tests"]),
    author="My Name",
    author_email="email@mycompany.com",
    description="My Back-End Server",
    entry_points={"console_scripts": [
                    "te-be=te.server:main"
                  ]},
    python_requires=">=3.7",
    install_requires=["connexion[swagger-ui]",
                      "Flask",
                      "gevent",
                      "redis",
                      "requests",
                      "proppkg @ https://site.mycompany.com/path/to/proppkg-70.1.0.tar.gz#egg=proppkg-70.1.0"
                      ],
    package_data={"te": ["openapi_te.yml"]},
    include_package_data=True,  # read MANIFEST.in
)
chrisinmtown
  • 3,571
  • 3
  • 34
  • 43
  • What `setuptools` version do you use? `python3 -c 'import setuptools; print(setuptools.__version__)'` – hoefling Sep 28 '21 at 20:24
  • Good question, I should have posted that. Running in my docker image `root@01c96a748bcf:/# python3 -c 'import setuptools; print(setuptools.__version__)'` returns 57.5.0 – chrisinmtown Sep 28 '21 at 21:30
  • Importing packages in the setup script may be another reason for bootstrapping issues. Can you show your `setup.py`? – hoefling Sep 29 '21 at 06:35
  • Another good question, thanks, I should have included `setup.py` in the question originally. It's there now. – chrisinmtown Sep 29 '21 at 12:40
  • Update a year later after I finally confirmed the proprietary package was definitely broken. Its `setup.py` file imported a file from its own package, and that file in turn imported `requests` which was not yet installed at that point. That explains the error during installation. I believe the PEP 508 solution should/would have worked fine. – chrisinmtown Nov 16 '22 at 18:19

0 Answers0