20

I have a C++ library that has a Python wrapper (written with SWIG). This library allows executing small user-defined code (a callback), such as element-wise operations on a vector. I.e. instead of just a + you can do whatever arbitrary binary function. Right now this is accomplished by accepting a callable Python object for the binary function and calling it. It works, but is about 80 times slower than code that doesn't have to bounce up and down into Python at every iteration.

How would I write/build/import a Cython function could be passed into my C++ library so that it can be called directly by the C++ library?

Edit: If I just stuck to C then I would write something like 

EWise(double (*callback)(double, double))

EWise would then callback(10, 20); or such. I want callback to be written in Cython, using whatever name the user wants, and a pointer to it has to be passed to my C++ library through Python somehow. That somehow is where I'm unclear.

Adam
  • 16,808
  • 7
  • 52
  • 98

3 Answers3

17

The trick with cython is in using the keyword public

cdef public double cython_function( double value, double value2 ):
    return value + value2

Then the command cythonize <your_file.pyx> along with <your_file.c> will create header <your_file.h> that you can include. Alternatively, you can create the header yourself:

#ifdef __cplusplus {
extern "C"
#endif

double cython_function( double value, double value2 );

#ifdef __cplusplus
}
#endif

Update:

Then with a little overlay from Python you can use ctypes's callback mechanism

func_type = CFUNCTYPE(c_double, c_double, c_double)

your_library.set_callback_function ( func_type(user_modules.cython_function) )
jesterjunk
  • 2,342
  • 22
  • 18
fabrizioM
  • 46,639
  • 15
  • 102
  • 119
  • That's what I want to accomplish, but since `cython_function` has to be user defined (not written by me), I won't know its name ahead of time. I need to support multiple such functions so I can't force a name either. – Adam Apr 19 '11 at 18:53
  • Do you know about numexpr ? It compiles python functions http://code.google.com/p/numexpr/ – fabrizioM Apr 19 '11 at 18:54
  • That looks interesting, I'll have to experiment with that. But my hands are tied for this project because our vectors are written in C++. It's a distributed memory system and it's got too many pieces to rip out its vectors and replace them with Python vectors :( – Adam Apr 19 '11 at 18:59
  • If you dont know the name of an object, how are you gonna be able to call it ? Post an example of what you want to achieve that explains a bit better the constraints. I think you would have to compile it at runtime, parse the source and get the name of the functions defined. – fabrizioM Apr 19 '11 at 19:45
2

You can achieve that by doing pure cdef functions :

# declare the prototype of your function
ctypedef void (*callback_ptr)(int arg)

# declare your function as cdef
cdef void my_callback(int arg):
    print 'doing some python here', arg

# now, you can use the cdef func as a callback
# in pure C !
cdef void run():
    cdef callback_ptr p = my_callback
    p(42)

if __name__ == '__main__':
    run()

Note: you can use "cython -a" to see that they are no python code involved for the content of run. So it will work with your c library.

jesterjunk
  • 2,342
  • 22
  • 18
tito
  • 12,990
  • 1
  • 55
  • 75
  • But I don't know that the callback is called `my_callback`. Remember that I don't write `my_callback`, my users do, and they need to be able to write any number of callback functions. – Adam Apr 19 '11 at 22:50
  • If your user write the callback in python, you have no way for doing that, except converting their python code to sort of C code. Using Cython, or any other parser tool. – tito Apr 19 '11 at 22:59
  • I'm interested in the case where my user writes a callback in Cython. They compile it. Using whatever Python or C++ trickery I want a pointer to their Cython function. – Adam Apr 19 '11 at 23:10
  • Ok, so if you don't even know the prototype of the user func, a func is still a void*. Use their Cython file directly, and do &func (for eg.) – tito Apr 19 '11 at 23:22
  • What do you mean use their Cython file directly? – Adam Apr 19 '11 at 23:25
  • If they provide a .pxd, you can do: from userfile cimport *. And they, you'll have their callback. (pyx = .c, pxd = .h) – tito Apr 19 '11 at 23:29
  • I'm writing a library with on the order of hundreds of users. That's not even remotely practical. – Adam Apr 19 '11 at 23:46
  • So at the end: You(C++) <- want c callback <- Python <- User Cython. I don't think you have a single chance of doing that. Cython->Python is generated. You might go Python->Cython by looking at the generation code of Cython/C, and do the inverse. But i'm sure it will not work. I think it's going out my scope, you're still in C, and C don't have introspection :) – tito Apr 20 '11 at 00:10
0

Embedding Python in Another Application may be useful reading.

jesterjunk
  • 2,342
  • 22
  • 18
Sean
  • 671
  • 5
  • 11
  • That means going through PyObject_CallObject() and friends to call the function, which is something I want to avoid. I want to call the C function directly. – Adam Apr 19 '11 at 01:47
  • Your biggest issue is you don't know how to do PyObject -> C right ? Cython + early declaration can help you no ? Declare your function in cython starting with cdef. You'll have a c definition, and you can still use Python in the C function. – tito Apr 19 '11 at 10:05
  • I can do PyObject -> C just fine. But that's slow. The point is to avoid building PyTuples and using PyObject_CallObject(). That's where the bulk of the overhead appears to be. – Adam Apr 19 '11 at 18:12
  • @tito, right I can write a C function in Python syntax with a cdef. But how do I get a pointer to that function so that my C++ code can call it directly, thus avoiding building argument tuples and other Python overhead? – Adam Apr 19 '11 at 18:15