Cython has "first class function objects" - how efficient are they? Do they have Python overhead?

Question

I have an external library that computes the optima, say minima, of a given function. Say its headers give me a function

double[] minimizer(ObjFun f)

where the headers define

typedef double (*ObjFun)(double x[])

I have generated Cython wrappers for this library. I now want to give user parameterized functions to it, specifically, I want to write a function

def getFunction(double q11, double q12, double q22):
    cdef f(double x[]):
        return x[0]*x[0]*q11 + 2*x[0]*x[1]*q12 +  x[1]*x[1]*q22
return f

that the user calls with their parameters (q11, q12, q22) to get a function that can be used as a callback for the C optimization library.

(The example above is contrived and simplified, the whole point of doing this in Cython is that I want nearly-C-efficient callbacks to give to the library.)

There is no way to do this in C, as observed by people in my other question. But in Cython I can compile this:

cdef class Quadratic:
    cdef double q11
    cdef double q12
    cdef double q22
    def __cinit__(self, double a, double b, double c):
        self.q11 = a
        self.q12 = b
        self.q22 = c
    cdef f(self, double x[]):
        return self.q11*x[0]*x[0] + self.q12*x[0]*x[1] + self.q22*x[1]*x[1]

(I have not yet tried using a generated function like this as an input to the library).

My question - is there Python overhead in this evaluation? I would like to have the parameterized function be nearly as efficient as if it were written in C.

If that is possible, how does Cython achieve this?

Answer 1

(Unless this is a very new addition to Cython then) this is not valid Cython code and can't be made to work. It fails for me with the error

C function definition not allowed here

The reason for this is that - as you already know - it is a genuine limitation of C: there is no valid C code that Cython could generate to create a closure. The reason being is that C would need to be able to dynamically generate a function at runtime to access the closured variables, and that isn't possible in standard C.

There are (at least) two three ways round this. First (and probably most sensible) you change your interface to accept a data pointer:

double[] minimizer(ObjFun f, void* data)

with

typedef double (*ObjFun)(double x[], void* data)

(where minimizer passes data to f.) If you're using an external library then you can't make this change, but I'd be surprised if it didn't use something like this - it's the standard solution.

The second way is to use ctypes to convert a Python callable into a C function pointer. This gets round the problem by dynamically generating machine code at runtime to match the C calling convention (and so can be blocked by security features that prohibit making memory executable). I show a basic example of how to do this in the second half of this answer. (I'm sure I have another more thorough answer showing this but I can't find it right now).

Using ctypes involves a reasonable amount of Python overhead, which is unavoidable.

(Edited addition) The third option to emulate a closure (somewhat unsatisfactorily) is to store the data in global variables:

# at global scope
q11 = 1.2
q22 = 3.2
q33 = 4.0

cdef double f(double x[]):
    # as before

The big limitation is that you can only use one version of f at once, so you can't run things in parallel or anything like that, but given the constraints that might be acceptable.

---

With respect to your cdef class version: calling it is pretty efficient but it effectively has a signature of:

`double (*)(Quadratic*, double [])`

so it won't fit the interface defined by your minimizer function.