Pass c function as argument to python function in cython

Question

I have the following code which works:

%%cython 

cdef int add(int  a, int b):
    return a+b

cdef int mult(int  a, int b):
    return a*b

ctypedef int (*function_type)(int  a, int b)

cdef int lambda_c(function_type func, int c, int d):
    return func(c,d)

print(lambda_c(add, 5, 6))
print(lambda_c(mult, 5, 6))

So I have lambda_c function that takes c function as an argument and I am not able to change it (as it it a wrapper over c library that is supported by another team).

What I want to do is to write a wrapper:

cdef class PyClass:
    def py_wrap(func, e, f):
        return lambda_c(func, e, f)

print(PyClass().py_wrap(add, 5, 5))
print(PyClass().py_wrap(mult, 6, 6))

But this throws an error:

Cannot convert Python object to 'function_type'

I also tried to cast func(return lambda_c(<function_type>func, e, f)) but got an error:

Python objects cannot be cast to pointers of primitive types

The idea behind this is following: any user will be able to write his own function in cython, compile it and then import and pass his function to PyClass().py_wrap method.

Is it even possible to import pure c function and pass it as a parameter with cython?

I also saw Pass cython functions via python interface but unlike the solution there I am not able to change lambda_c functon and turn it into a class. Moreover lambda_c takes only functions of certain type (function_type in our example)

Answer 1

If you want to pass an arbitrary Python callable as a C function pointer then it doesn't work - there's no way to do this in standard C (and thus it's impossible to for Cython to generate the code). There's a very hacky workaround involving ctypes (which can do runtime code generation which I'll find links to if needed. However I don't really recommend it.

If you're happy for your users to write cdef functions in Cython (the question implies you are) then you can just build on my answer to your question yesterday.

1. Write a suitable wrapper class (you just need to change the function pointer type) - this gets split between your .pxd and .pyx files that you write.

2. Have your users cimport it and then use it to expose their cdef classes to Python:

   from your_module cimport FuncWrapper
   
   cdef int add_c_implementation(int  a, int b):
       return a+b
   
   # `add` accessible from Python
   add = WrapperFunc.make_from_ptr(add_c_implementation)
   

3. Change PyClass to take FuncWrapper as an argument:

   # this is in your_module.pyx
   
   cdef class PyClass:
       def py_wrap(FuncWrapper func, e, f):
           return lambda_c(func.func, e, f)
   

4. Your users can then use their compiled functions from Python:

   from your_module import PyClass
   from users_module import add
   
   PyClass().py_wrap(add,e,f)
   

---

Really all this is doing is using a small Python wrapper to allow you to pass around a type that Python normally can't deal with. You're pretty limited in what it's possible to do with these wrapped function pointers (for example they must be set up in Cython) but it does give a handle to select and pass them.

Answer 2

I am not sure if you are allowed to change the function pointer type from

ctypedef int (*function_type)(int  a, int b)

to

ctypedef int (*function_type)(int a, int b, void *func_d)

but this is usually the way callback functions are implemented in C. void * parameter func_d to the function contains the user-provided data in any form. If the answer is yes, then you can have the following solution.

First, create the following definition file in Cython to reveal your C API to other Cython users:

# binary_op.pxd
ctypedef int (*func_t)(int a, int b, void *func_d) except? -1

cdef int func(int a, int b, void *func_d) except? -1

cdef class BinaryOp:
  cpdef int eval(self, int a, int b) except? -1

cdef class Add(BinaryOp):
  cpdef int eval(self, int a, int b) except? -1

cdef class Multiply(BinaryOp):
  cpdef int eval(self, int a, int b) except? -1

This basically allows any Cython user to cimport these definitions directly into their Cython code and bypass any Python-related function calls. Then, you implement the module in the following pyx file:

# binary_op.pyx
cdef int func(int a, int b, void *func_d) except? -1:
  return (<BinaryOp>func_d).eval(a, b)

cdef class BinaryOp:
  cpdef int eval(self, int a, int b) except? -1:
    raise NotImplementedError()

cdef class Add(BinaryOp):
  cpdef int eval(self, int a, int b) except? -1:
    return a + b

cdef class Multiply(BinaryOp):
  cpdef int eval(self, int a, int b) except? -1:
    return a * b

def call_me(BinaryOp oper not None, c, d):
  return func(c, d, <void *>oper)

As you can see, BinaryOp serves as the base class which raises NotImplementedError for its users who do not implement eval properly. cpdef functions can be overridden by both Cython and Python users, and if they are called from Cython, efficient C calling mechanisms are involved. Otherwise, there is a small overhead when called from Python (well, of course, these functions work on scalars, and hence, the overhead might not be that small).

Then, a Python user might have the following application code:

# app_1.py
import pyximport
pyximport.install()

from binary_op import BinaryOp, Add, Multiply, call_me

print(call_me(Add(), 5, 6))
print(call_me(Multiply(), 5, 6))

class LinearOper(BinaryOp):
  def __init__(self, p1, p2):
    self.p1 = p1
    self.p2 = p2
  def eval(self, a, b):
    return self.p1 * a + self.p2 * b

print(call_me(LinearOper(3, 4), 5, 6))

As you can see, they can not only create objects from efficient Cython (concrete) classes (i.e., Add and Multiply) but also implement their own classes based on BinaryOp (hopefully by providing the implementation to eval). When you run python app_1.py, you will see (after the compilations):

11
30
39

Then, your Cython users can implement their favorite functions as follows:

# sub.pyx
from binary_op cimport BinaryOp

cdef class Sub(BinaryOp):
  cpdef int eval(self, int a, int b) except? -1:
    return a - b

Well, of course, any application code that uses sub.pyx can use both libraries as follows:

import pyximport
pyximport.install()

from sub import Sub
from binary_op import call_me

print(call_me(Sub(), 5, 6))

When you run python app_2.py, you get the expected result: -1.

EDIT. By the way, provided that you are allowed to have the aforementioned function_type signature (i.e., the one that has a void * parameter as the third argument), you can in fact pass an arbitrary Python callable object as a C pointer. For this to happen, you need to have the following changes:

# binary_op.pyx
cdef int func(int a, int b, void *func_d) except? -1:
  return (<object>func_d)(a, b)


def call_me(oper not None, c, d):
  return func(c, d, <void *>oper)

Note, however, that Python now needs to figure out which object oper is. In the former solution, we were constraining oper to be a valid BinaryOp object. Note also that __call__ and similar special functions can only be declared def, which limits your use case. Nevertheless, with these last changes, we can have the following code run without any problems:

print(call_me(lambda x, y: x - y, 5, 6))

Answer 3

Thanks to @ead , I have changed code a bit and the result satisfies me:

cdef class PyClass:
    cdef void py_wrap(self, function_type func, e, f):
        print(lambda_c(func, e, f))

PyClass().py_wrap(mult, 5, 5)

For my purposes it is ok to have void function, but I do not know, how to make it all work with method, that should return some value. Any ideas for this case will be useful

UPD: cdef methods are not visible from python, so it looks like there is no way to make things work