17

I have a Python project that uses setuptools for deployment and I mostly followed this guide regarding project structure. The project uses Google Protocol Buffers to define a network message format. My main issue is how to make setup.py call the protoc-compiler during installation to build the definitions into a _pb2.py file.

In this question the advice was given to just distribute the resulting _pb2.py files along with the project. While this might work for very similar platforms, I've found several cases where this does not work. For example, when I develop on a Mac that uses Anaconda Python and copy the resulting _pb2.py along with the rest of the project to a Raspberry Pi running Raspbian, there are always import errors coming from the _pb2.py modules. However, if I compile the .proto files freshly on the Pi, the project works as expected. So, distributing the compiled files does not seem like an option.

Kind of looking for working and best practice solutions here. It can be assumed that the protoc-compiler is installed on the target platform.

Edit:

Since people ask for the reasons of the failure. On the Mac, the protobuf version is 2.6.1. and on the Pi it's 2.4.1. Apparently, the internal API as used by the generated protoc compiler output has changed. The output is basically:

  File "[...]network_manager.py", line 8, in <module>
      import InstrumentControl.transports.serial_bridge_protocol_pb2 as protocol
  File "[...]serial_bridge_protocol_pb2.py", line 9, in <module>
      from google.protobuf import symbol_database as _symbol_database
  ImportError: cannot import name symbol_database
Community
  • 1
  • 1
jan
  • 927
  • 2
  • 9
  • 20
  • I had similar issues, and always compiled on the PI. The problem stems AFAIR from different protobuf-versions. So - which versions do you have on the respective machines, and can't you upgrade them to the same? – deets Jan 08 '15 at 15:24
  • I'd really like to avoid that. I can (probably) do that on the machines that I control but it does not feel like the right way to do it. And it probably fails when distributing the software to others. Making them install a specific version of protocol buffers when their current version could (by re-compiling) be used, does not feel right. – jan Jan 08 '15 at 15:26
  • Did you researched - what causes import errors? I would choose packaging _pb2.py files strategy and researching import error problems. – Gill Bates Jan 08 '15 at 15:28
  • There are always minimal requirements to fulfill for a user. So either you ask them to install certain software, or you bundle it within your application. You could pick a certain version of protocol buffers and include the whole stack with your product. Just ensure it's picked up first when importing it. – deets Jan 08 '15 at 15:30
  • @deets: I respect that opinion, it would certainly make my life easier. But that specific protobuf version is not really a requirement if I could just call protoc from the setup script. And having a specific pinned version as a requirement could be even more problematic in the future when the internal API changes again and suddenly users require an older version to run. I'd really like to avoid that support hell. So let's just assume that distributing the _pb2.py files is not a practical option. – jan Jan 08 '15 at 15:47
  • Well, you can't assume your way out of the reality of protobuf: it's backwards-compatible wrt to *using* older generated _pb2.py-files, but will always only generate files compatible with the version protoc has. So if you want to run protoc on various machines/platforms, you have to ensure it's the same version everywhere. There are several options there, I'd opt for including it in your app. Or you always use one dedicated machine (for me, it was the PI as I didn't want to upgrade it's package) and just make a minimum requirement for exactly that version. But I fail to see anything else. – deets Jan 08 '15 at 17:41

3 Answers3

18

Ok, I solved the issue without requiring the user to install a specific old version or compile the proto files on another platform than my dev machine. It's inspired by this setup.py script from protobuf itself.

Firstly, protoc needs to be found, this can be done using

# Find the Protocol Compiler.
if 'PROTOC' in os.environ and os.path.exists(os.environ['PROTOC']):
  protoc = os.environ['PROTOC']
else:
  protoc = find_executable("protoc")

This function will compile a .proto file and put the _pb2.py in the same spot. However, the behavior can be changed arbitrarily.

def generate_proto(source):
  """Invokes the Protocol Compiler to generate a _pb2.py from the given
  .proto file.  Does nothing if the output already exists and is newer than
  the input."""

  output = source.replace(".proto", "_pb2.py")

  if (not os.path.exists(output) or
      (os.path.exists(source) and
       os.path.getmtime(source) > os.path.getmtime(output))):
    print "Generating %s..." % output

    if not os.path.exists(source):
      sys.stderr.write("Can't find required file: %s\n" % source)
      sys.exit(-1)

    if protoc == None:
      sys.stderr.write(
          "Protocol buffers compiler 'protoc' not installed or not found.\n"
          )
      sys.exit(-1)

    protoc_command = [ protoc, "-I.", "--python_out=.", source ]
    if subprocess.call(protoc_command) != 0:
      sys.exit(-1)

Next, the classes _build_py and _clean are derived to add building and cleaning up the protocol buffers.

# List of all .proto files
proto_src = ['file1.proto', 'path/to/file2.proto']

class build_py(_build_py):
  def run(self):
    for f in proto_src:
        generate_proto(f)
    _build_py.run(self)

class clean(_clean):
  def run(self):
    # Delete generated files in the code tree.
    for (dirpath, dirnames, filenames) in os.walk("."):
      for filename in filenames:
        filepath = os.path.join(dirpath, filename)
        if filepath.endswith("_pb2.py"):
          os.remove(filepath)
    # _clean is an old-style class, so super() doesn't work.
    _clean.run(self)

And finally, the parameter

cmdclass = { 'clean': clean, 'build_py': build_py }

needs to be added to the call to setup and everything should work. Still have to check for possible quirks, but so far it works flawlessly on the Mac and on the Pi.

jan
  • 927
  • 2
  • 9
  • 20
4

I've just started protobuf-setuptools package to use the most sane part of this code. It still needs improvements, so any feedback is welcome!

Check it out: https://pypi.python.org/pypi/protobuf-setuptools

pupssman
  • 541
  • 2
  • 11
3

Another solution is to bundle the protobuf library with your app, rather than use the installed version on the target machine. This way you know there's no version mismatch with your generated code.

Kenton Varda
  • 41,353
  • 8
  • 121
  • 105