16

I am working on an existing Python 3 code-base that provides a setup.py so the code is installed as a Python library. I am trying to get this internal library installed with its own dependencies (the usual data science ones e.g. pandas, pyodbc, sqlalchemy etc).

I would like to have this internal library to deal with these dependencies and assume that if that library is installed, then all the transitive dependencies are assumed to be installed. I also would like to have the Anaconda (conda) version of the package rather than the pip version.

I started with a requirements.txt, but moved quickly to this field in setup.py:

  install_requires=[
      "pyodbc>=4.0.27",
      "sqlalchemy>=1.3.8",
      "pandas>=0.25.1",
      "requests>=2.22.0",
      "assertpy>=0.14",
      "cycler>=0.10.0",
  ]

However when I run the installation process:

  • either with python setup.py install --record installed_files.txt
  • or with pip install .

I see that there is some gcc / C++ compilation going on that shows logs about Python wheels (I don't completely understand the implications of Python eggs and Python wheels, but AFAIK if conda is available then I should go with the conda version rather than eggs/wheels because then I don't have to take care of the C++ code underneath the Python code).

I really would prefer having conda to install these C++ blobs wrapped in some Python code as a libraries e.g. pandas.

  • is it possible at all to have conda driving the installation process described in setup.py so I am not dealing with gcc?
  • how can I make sure that other Python code depending on this internal library (installed via setup.py) is using the same (transitive) dependencies defined in that setup.py?

Regardless the installation method, how can I make sure that the dependencies for e.g. pandas are installed as well? Sometimes I see that numpy as a dependency of pandas is not installed when running setup.py, but I would like to avoid doing this manually (e.g. with some requirements.txt file).

TPPZ
  • 4,447
  • 10
  • 61
  • 106

1 Answers1

5

pip doesn't know about conda, so you cannot build a pip-installable package that pulls in its dependencies from conda channels.

conda doesn't care about setup.py, it uses a different format for recording dependencies.

To install your code with conda, you should create a conda package, and specify your dependencies in a meta.yaml file. Refer to the documentation of "conda build" for details.

https://docs.conda.io/projects/conda-build/en/latest/resources/define-metadata.html

Roland Weber
  • 3,395
  • 12
  • 26
  • 1
    I think this can get confusing because so many conda packages are basically wrappers around a `pip install` build script so lots of conda packages do have a `setup.py` in addition to a recipe. e.g. https://github.com/conda-forge/requests-feedstock/blob/master/recipe/meta.yaml – medley56 Jul 15 '20 at 19:07
  • 1
    While this may have been accepted it is not an answer to the question in the title. Change the title and I'll change my vote to up if appropriate. – user3673 Oct 26 '20 at 21:21
  • 1
    @user3673 The title is chosen by the person asking the question, not by those writing answers. – Roland Weber Oct 30 '20 at 14:30