2

Copy files to pip's build directory

(Nasty) use-case: I have a 3rd party subproject that I have to endow with setup.py, so that it would run make producing certain artifacts that I want to list in scripts=. I've derived a class from setuptools.install.install that chdir()s into a folder adjacent to __file__ and makes. However, when run with pip install, pip would only copy what seems to be source files to the build directory. I've tried specifying data_files=, but that didn't help.

How can I hint pip to copy specific files to the build dir?

P.S. I know it may look like a wrong problem to solve, but I do specifically want to copy those files to build dir, to make and to add artifacts to the wheel.

UPD: in reply to the comment on setup.py MVE.

Example project structure:

├── libs
│   └── external
│       ├── Makefile
│       └── src
│           └── _...
├── setup.py
└── src
    └── __init__.py

Example setup.py:

from setuptools import setup
from setuptools.command.install import install
from subprocess import Popen
import os
import glob


EXT_DIR = os.path.join(os.path.dirname(__file__), 'libs', 'external')
class RunMake(install):
    """Makefile on setuptools install."""
    def run(self):
        old_dir = os.getcwd()
        try:
            os.chdir(EXT_DIR)
            self.spawn(['make'])
        finally:
            os.chdir(old_dir)
        install.run(self)




setup(name='example',
      version='0.0.1',
      packages=['example'],
      package_dir={'example': 'src'},
      cmdclass={'install': RunMake},
      scripts=['libs/external/build/bin/SomeBinary'],
      data_files=[
          ('libs', glob.glob('libs/**'))
          ]
      )

UPD: MANIFEST.in

Tried out just this, and Makefile still isn't there

include libs/external/Makefile
include libs/external/*

The resulting .egg-info:

setup.py
poisson_reconstruction.egg-info/PKG-INFO
poisson_reconstruction.egg-info/SOURCES.txt
poisson_reconstruction.egg-info/dependency_links.txt
poisson_reconstruction.egg-info/top_level.txt
src/__init__.py
Serge Kozlukov
  • 21
  • 3
  • Not sure what the issue is exactly. Are the files not correctly generated or not correctly packaged (i.e. either not added to the distributions or not installed)? Maybe show your `setup.py`? Might the following help? https://stackoverflow.com/a/54953494/11138259 – sinoroc Nov 13 '19 at 20:27
  • @sinoroc updated the question with an example – Serge Kozlukov Nov 13 '19 at 20:58
  • For instance, `Makefile` won't be there, when pip calls `setup.py install` – Serge Kozlukov Nov 13 '19 at 21:00
  • `data_files` is probably wrong for your use case. Looks like you'd be better off adding the whole `libs` directory tree to the `MANIFEST.in` file (assuming you only want it in the _sdist_ but not in the _bdist_). – sinoroc Nov 13 '19 at 21:06
  • Yeah, I already figured `data_files` is not for me, just couldn't find the right thing in meaningful time. And I've tried `MANIFEST.in` too, still not working – Serge Kozlukov Nov 13 '19 at 21:17
  • Could you add the `MANIFEST.in` to the question? Find the `SOURCES.txt` file in the `*.egg-info` directory and make sure it lists all the files you need for the custom build step (`make`). – sinoroc Nov 13 '19 at 21:21
  • @sinoroc updated – Serge Kozlukov Nov 13 '19 at 21:30
  • 1
    Aha, I forgot to wipe `*.egg-info` – Serge Kozlukov Nov 13 '19 at 21:34
  • When in doubt, make sure to delete the `.egg-info` directory after every modification of the packaging (`MANIFEST.in` in particular, and `setup.py`) and before executing `python setup.py sdist` then looking at the content of the `SOURCES.txt` file. – sinoroc Nov 13 '19 at 21:34

0 Answers0