5

My python package depends on a private package that is not in pypi. You can assume that the structure of the private package is this.

Goal: store the 3rd party private package somewhere within my own package in a whatever way such that when I run pip install -r requirements.txt (or even better pip install <my_pkg_name>), the 3rd party dependency is installed. This must work in Windows and Linux.

A potential workaround: Do not install the 3rd party module. Instead, place it in your main package directory as it is and import its modules.

/my_pkg
    /non_pypi_pkg
    /my_pkg
        module.py

Where module.py is able to import non_pypi_pkg without installing non_pypi_pkg.

Question: What what is the recommended practice to achieve the goal? Can we do any better than my workaround?

MLguy
  • 1,776
  • 3
  • 15
  • 28
  • 1
    not sure of the context, but it's possible to set up your own local pypi mirror, and serve your own content there. maybe see https://stackoverflow.com/questions/1235331/how-to-roll-my-own-pypi – Corley Brigman Mar 27 '18 at 14:05
  • Good point. I've already maintained a local pypi mirror in the past, but I'd like to use a simpler solution in this case. – MLguy Mar 27 '18 at 14:12
  • No matter what option you choose, you need to write a `setup.py` for the private package. Then, setting up a private PyPI repo is one option, another would be installing from URL via `dependency_links` or even installing directly from VCS, again via `dependency_links`. `pip` can also handle VCS URLs, i.e. `pip install git+https://github.com/user/project@branch`. – hoefling Mar 27 '18 at 18:15

1 Answers1

2

When I work with third-party packages, I often use one of the next methods using setuptools.

1. Install directly indicating a URL path in setup.py.

Having the following structure:

/my_pkg
├── /my_pkg
│   ├── __init__.py
│   └── module.py
├── setup.py

setup.py

setup(
    name="my_pkg",
    version="0.0.0",
    packages=find_packages(),
    install_requires=['flask', 
        'non_pypi_pkg @ https://github.com/<user>/non_pypi_pkg/archive/main.zip#egg=non_pypi_pkg-0.0.1',
        ]
)

When the repository is private I use a personal access token (by github), in the same example could be pkg @ https://<token>@github.com/<user>/pkg/...

  • Then pip install -e my_pkg

2. Working from a monorepo with multiple packages to enforce python separation layers.

Having the following structure:

/
├── my_pkg
│   ├── my_pkg
│   │   ├── __init__.py
│   │   └── module.py
│   └── setup.py
├── non_pypi_pkg
│   ├── non_pypi_pkg
│   │   ├── __init__.py
│   │   └── module_b.py
│   └── setup.py

/non_pypi_pkg/setup.py

setup(
    name="non_pypi_pkg",
    version="0.0.1",
    packages=find_packages(),
    install_requires=[]
)

/my_pkg/setup.py

setup(
    name="my_pkg",
    version="0.0.0",
    packages=find_packages(),
    install_requires=['flask', 'non_pypi_pkg']
)

  • Here it is necessary to install in order of dependence. For example:

     pip install -e non_pypi_pkg; pip install -e my_pkg;

I prefer to use a script as a Makefile for these tasks but is possible too with a bash or power-shell script like as next:

#!/bin/bash
libraries=(non_pypi_pkg my_pkg)

for dirname in $(ls); do
    if [ -d "$dirname" ] && [ -e "$dirname/setup.py" ]; then
        pip install -e $dirname
    fi
done
Henry Palacios
  • 301
  • 3
  • 8