8

I have some protocol buffer definitions which need to be built to Python source as part of the pip install process. I've subclassed the setuptools.command.install command in setup.py but I think it's trying to run the Makefile after the package is installed so the sources aren't recognised.

I can't find information about what happens during a pip installation. Can anyone shed any light?

setup.py:

import subprocess
import sys

from setuptools import setup
from setuptools.command.install import install

class Install(install):
    """Customized setuptools install command - builds protos on install."""
    def run(self):
        protoc_command = ["make", "python"]
        if subprocess.call(protoc_command) != 0:
            sys.exit(-1)
        install.run(self)


setup(
    name='myprotos',
    version='0.0.1',
    description='Protocol Buffers.',
    install_requires=[],
    cmdclass={
        'install': Install,
    }
)

Output of $ pip install -vvv .:

Processing /path/to/myprotos
  Running setup.py (path:/private/var/folders/3t/4qwkfyr903d0b7db7by2kj6r0000gn/T/pip-jpgCby-build/setup.py) egg_info for package from file:///path/to/myprotos
    Running command python setup.py egg_info
    running egg_info
    creating pip-egg-info/myprotos.egg-info
    writing pip-egg-info/myprotos.egg-info/PKG-INFO
    writing top-level names to pip-egg-info/myprotos.egg-info/top_level.txt
    writing dependency_links to pip-egg-info/myprotos.egg-info/dependency_links.txt
    writing manifest file 'pip-egg-info/myprotos.egg-info/SOURCES.txt'
    reading manifest file 'pip-egg-info/myprotos.egg-info/SOURCES.txt'
    writing manifest file 'pip-egg-info/myprotos.egg-info/SOURCES.txt'
  Source in /private/var/folders/3t/4qwkfyr903d0b7db7by2kj6r0000gn/T/pip-jpgCby-build has version 0.0.1, which satisfies requirement myprotos==0.0.1 from file:///path/to/myprotos
Building wheels for collected packages: myprotos
  Running setup.py bdist_wheel for myprotos: started
  Destination directory: /var/folders/3t/4qwkfyr903d0b7db7by2kj6r0000gn/T/tmpD7dfGKpip-wheel-
  Running command /usr/local/opt/python/bin/python2.7 -u -c "import setuptools, tokenize;__file__='/private/var/folders/3t/4qwkfyr903d0b7db7by2kj6r0000gn/T/pip-jpgCby-build/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /var/folders/3t/4qwkfyr903d0b7db7by2kj6r0000gn/T/tmpD7dfGKpip-wheel- --python-tag cp27
  running bdist_wheel
  running build
  installing to build/bdist.macosx-10.12-x86_64/wheel
  running install

  # THIS IS MY MAKEFILE RUNNING
  Grabbing github.com/google/protobuf...
  Building Python protos...
  # MAKEFILE COMPLETE

  running install_egg_info
  running egg_info
  creating myprotos.egg-info
  writing myprotos.egg-info/PKG-INFO
  writing top-level names to myprotos.egg-info/top_level.txt
  writing dependency_links to myprotos.egg-info/dependency_links.txt
  writing manifest file 'myprotos.egg-info/SOURCES.txt'
  reading manifest file 'myprotos.egg-info/SOURCES.txt'
  writing manifest file 'myprotos.egg-info/SOURCES.txt'
  Copying myprotos.egg-info to build/bdist.macosx-10.12-x86_64/wheel/myprotos-0.0.1-py2.7.egg-info
  running install_scripts
  creating build/bdist.macosx-10.12-x86_64/wheel/myprotos-0.0.1.dist-info/WHEEL
  Running setup.py bdist_wheel for myprotos: finished with status 'done'
  Stored in directory: /Users/jds/Library/Caches/pip/wheels/92/0b/37/b5a50146994bc0b6774407139f01d648ba3a9b4853d2719c51
  Removing source in /private/var/folders/3t/4qwkfyr903d0b7db7by2kj6r0000gn/T/pip-jpgCby-build
Successfully built myprotos
Installing collected packages: myprotos
  Found existing installation: myprotos 0.0.1
    Uninstalling myprotos-0.0.1:
      Removing file or directory /usr/local/lib/python2.7/site-packages/myprotos-0.0.1.dist-info/DESCRIPTION.rst
      Removing file or directory /usr/local/lib/python2.7/site-packages/myprotos-0.0.1.dist-info/INSTALLER
      Removing file or directory /usr/local/lib/python2.7/site-packages/myprotos-0.0.1.dist-info/METADATA
      Removing file or directory /usr/local/lib/python2.7/site-packages/myprotos-0.0.1.dist-info/RECORD
      Removing file or directory /usr/local/lib/python2.7/site-packages/myprotos-0.0.1.dist-info/WHEEL
      Removing file or directory /usr/local/lib/python2.7/site-packages/myprotos-0.0.1.dist-info/metadata.json
      Removing file or directory /usr/local/lib/python2.7/site-packages/myprotos-0.0.1.dist-info/top_level.txt
      Successfully uninstalled myprotos-0.0.1

Successfully installed myprotos-0.0.1
Cleaning up...

Should my Makefile be running early in the process to generate the source files? Do the files need to be there before egg_info runs for example?

If I manually run the Makefile and then install the package then it works.

Update

Here is the structure of my project:

myprotos
├── Makefile
├── README.md
├── document.proto
├── myprotos         # Generated by Makefile
│   ├── __init__.py  # Generated by Makefile
│   └── proto_pb2.py # Generated by Makefile
└── setup.py

Here is the section of the Makefile which generates the Python source from Potocol Buffer definitions:

python: protoc deps
    # the protoc and deps command above just downloads
    # the `protoc` binary to a local bin directory 
    @echo "Building Python protos..."
    @mkdir -p "${PYTHON_OUT}"
    @touch "${PYTHON_OUT}"/__init__.py
    @printf "__all__ = ['proto_pb2']" > "${PYTHON_OUT}"/__init__.py
    @PATH="${LOCAL_BINARY_PATH}:$$PATH" protoc \
        --proto_path="${BASE}" \
        --proto_path="${GOPATH}/src/github.com/google/protobuf/src" \
        --python_out="${PYTHON_OUT}/" \
        ${PROTOS}
Josh
  • 3,445
  • 5
  • 37
  • 55
  • You should run it [on build](https://stackoverflow.com/a/14644982/636626), not on install – Nils Werner Jul 27 '17 at 10:12
  • @NilsWerner I've updated `setup.py` to subclass the build command but I still have problems importing the package. There seems to be no sources in the wheel that's built. – Josh Jul 27 '17 at 10:23
  • What does your [`MANIFEST.in`](https://docs.python.org/2/distutils/sourcedist.html#manifest-template) look like? Also, your `Makefile` generates source files that should go in the `sdist`? – Nils Werner Jul 27 '17 at 10:45
  • @NilsWerner I don't have a `MANIFEST.in` file. However, if I run `make` and then `pip install .` then the package is installed and useable. It seems that even in `build` the Python source isn't being generated early enough. – Josh Jul 27 '17 at 10:53
  • yes, that's expected. You need to add all non-py source files to `MANIFEST.in` and amend your question with what exactly your `Makefile` does and what files it consumes and generates. Because if it generates source files, it needs to go in the `sdist` command, if it compiles existing source files it needs to go in the `build` section. – Nils Werner Jul 27 '17 at 11:00
  • @NilsWerner ok right. Please see my update for the project structure and the makefile. – Josh Jul 27 '17 at 11:09
  • OK, I would say you need to run `make` in `build`, add `Makefile` and `doctument.proto` to `MANIFEST.in` (so they end up in the `sdist`), and add a field `packages=['myprotos']` to `setup()`. – Nils Werner Jul 27 '17 at 11:15
  • GENIUS! Good work @NilsWerner, makes perfect sense. I just needed to ensure the `myprotos` module directory is created before `egg_info` runs, otherwise it fails. Do you want to try and put together an answer so I can give you those wholesome internet points? – Josh Jul 27 '17 at 11:25

1 Answers1

8

OK, there are three things you need to change here:

  1. Add Makefile and document.proto to a new file MANIFEST.in.

     Makefile
 document.proto

    If you do that, the .zip file created by python setup.py sdist (which is also uploaded to PyPI) will contain those files.

  2. You need to run your make command during python setup.py build, not during install. Since you are generating Python code, you will need to change the build_py command here:

     import sys
 import subprocess

 from setuptools import setup
 from setuptools.command.build_py import build_py

 class Build(build_py):
     """Customized setuptools build command - builds protos on build."""
     def run(self):
         protoc_command = ["make", "python"]
         if subprocess.call(protoc_command) != 0:
             sys.exit(-1)
         super().run()


 setup(
     name='buildtest',
     version='1.0',
     description='Python Distribution Utilities',
     packages=['buildtest'],
     cmdclass={
         'build_py': Build,
     }
 )

    If your Makefile generated machine code, i.e. from C or any other compiled language, you should change the build_ext command:

     import sys
 import subprocess

 from setuptools import setup
 from setuptools.command.build_ext import build_ext

 class Build(build_ext):
     """Customized setuptools build command - builds protos on build."""
     def run(self):
         protoc_command = ["make", "python"]
         if subprocess.call(protoc_command) != 0:
             sys.exit(-1)
         super().run()


 setup(
     name='buildtest',
     version='1.0',
     description='Python Distribution Utilities',
     packages=['buildtest'],
     has_ext_modules=lambda: True,
     cmdclass={
         'build_ext': Build,
     }
 )

  3. Lastly, you need to tell setuptools to install the resulting package on install by defining an attribute packages in setup():

     setup(
     ...
     packages=['myprotos']
 )

The reason for deciding to run build_py or build_ext lies in the situation when the two are run:

  • build_py is also run when creating source distributions, which have to be cross-platform. Compiled extensions are usually not cross-platform, so you cannot compile them in this step.
  • build_ext is only run when you are creating binary distributions, which are platform-specific. It is OK to compile to platform-specific machine code here.
Nils Werner
  • 34,832
  • 7
  • 76
  • 98