Compile file with two separate libraries in Cython

Question

I wrote a library in Cython that has two different "modes":

If rendering, I compile using GLFW.
If not rendering, I compile using EGL, which is faster, but I have not figured out how to render with it.

What is the recommended way to handle this situation?

Right now, I have the following directory structure:

mujoco
├── __init__.py
├── simEgl.pyx
├── simGlfw.pyx
├── sim.pxd
└── sim.pyx

simEgl.pyx contains EGL code and simGlfw.pyx contains GLFW code. setup.py uses an environment variable to choose one or the other for the build.

This works ok, except that I need to recompile the code every time I want to switch between modes. There must be a better way.

Update

I agree that the best approach is to simultaneously compile two different libraries and use a toggle to choose which one to import. I already do have a base class in sim.pyx with shared functionality. However this base class must itself be compiled with the separate libraries. Specifically, sim.pyx depends on libmujoco.so which depends on either GLFW or EGL.

Here is my exhaustive search of possible approaches:

If I do not compile an extension for sim.pyx, I get ImportError: No module named 'mujoco.sim'
If I compile an extension for sim.pyx without including graphics libraries in the extension, I get ImportError: /home/ethanbro/.mujoco/mjpro150/bin/libmujoco150.so: undefined symbol: __glewBlitFramebuffer
If I compile an extension for sim.pyx and choose one set of graphics libraries (GLFW), then when I try to use the other set of graphics libraries (EGL) this does not work either unsurprisingly: ERROR: GLEW initalization error: Missing GL version
If I compile two different versions of the sim.pyx library, one with one set of libraries, one with the other, I get: TypeError: unorderable types: dict() < dict() which is not a very helpful error message, but appears to result from trying to share a source file between two different extensions.

Something like option 4 should be possible. In fact, if I were working in raw C, I would simply build two shared objects side by side using the different libraries. Any advice on how to get around this Cython limitation would be very welcome.

You probably want to just build both as separate extension modules? e.g. https://stackoverflow.com/a/21826294/3657742 as an example — chrisb, Dec 01 '17 at 21:05
I largely agree with @chrisb. You write two extension modules with identical interfaces and use a `if rendering: import simGlfw as s; else: import simEgl as s`. That might change if you want to use Cython (i.e. compile-time) features but if that's the case you need to give more detail (a common base class might be a good solution though) — DavidW, Dec 02 '17 at 11:23
I agree that this is the way to go. However, it does require compiling `sim.pyx` file into two different `.so` files. I tries doing this with `Extension(name='sim1', ...)` and `Extension(name='sim2', ...) ` but this caused an Import Error. Perhaps you can suggest a better way? — ethanabrooks, Dec 03 '17 at 00:32
@chrisb, the critical difference between my situation and the post you referenced is that that post does not need the same source in two separate extensions. — ethanabrooks, Dec 04 '17 at 20:01
@ethanabrooks (Not 100% sure but...) Cython/Python modules get their name from the source filename, which I think is what is causing you problems. What you could try is creating two almost empty .pyx files with the correct name, and use [Cython's little used include mechanism](http://cython.readthedocs.io/en/latest/src/userguide/language_basics.html#the-include-statement) to load the shared code in each of them. — DavidW, Dec 05 '17 at 19:27
@DavidW that worked. Post an answer so that I can give you credit. — ethanabrooks, Dec 06 '17 at 18:04

score 0 · Accepted Answer · answered Dec 06 '17 at 18:35

(This answer is just a summary of the comments with a bit more explanation.)

My initial suggestion was to create two extension modules defining a common interface. That way you pick which to import in Python but be able to use them in the same way once imported:

if rendering:
   import simGlfw as s
else:
   import simEgl as s
s.do_something() # doesn't matter which you imported

It appears from the comments that the two modules also share a large chunk of their code and its really just the library that they're linked with that defines how they behave. Trying to re-use the same sources with

Extension(name='sim1', sources=["sim.pyx",...)
Extension(name='sim2', sources=["sim.pyx",...)

fails. This is because Cython assumes that the module name will be the same as the filename, and so creates a function PyInit_sim (on Python 3 - Python 2 is named slightly differently but the idea is the same). However, when you import sim1.so it looks for the function PyInit_sim1, fails to find it, and gives an error.

An easy way round it is to put the common code in "sim.pxi" and use Cython's largely obsolete include mechanism to textually include that code in sim1.pyx and sim2.pyx

include "sim.pxi"

Although include is generally no longer recommended and cimport is preferred since it provides more "Python-like" behaviour, include is a simple solution to this particular problem.

Compile file with two separate libraries in Cython

Update

1 Answers1