Pass cython functions via python interface

Question

Can a cdef Cython function be passed to another (python def) cython function from a Python script?

Minimal example:

test_module.pyx

cpdef min_arg(f, int N):
    cdef double x = 100000.
    cdef int best_i = -1

    for i in range(N):
        if f(i) < x:
            x = f(i)
            best_i = i
    return best_i

def py_f(x):
    return (x-5)**2

cdef public api double cy_f(double x):
    return (x-5)**2

test.py

import pyximport; pyximport.install()
import testmodule

testmodule.min_arg(testmodule.py_f, 100)

This works well, but I want to be able to also do

testmodule.min_arg(testmodule.cy_f, 100)

from a test.py, to have cython's speed (no Python overhead for each f(i) call). But obviously, Python doesn't know about cy_f, because it's not def or cpdef declared.

I was hoping something like this existed:

from scipy import LowLevelCallable
cy_f = LowLevelCallable.from_cython(testmodule, 'cy_f')
testmodule.min_arg(cy_f, 100)

But this gives TypeError: 'LowLevelCallable' object is not callable.

Thank you in advance.

score 1 · Accepted Answer · answered Sep 04 '17 at 15:13

The LowLevelCallable is a class of functions that must be accepted by the underlying Python module. This work has been done for a few modules, including the quadrature routine scipy.integrate.quad

If you wish to use the same wrapping method, you must either go through the SciPy routines that make use of it, such as scipy.ndimage.generic_filter1d or scipy.integrate.quad. The code sits in compiled extensions, however.

The alternative, if your problem is reasonably well defined for the callback, is to implement this yourself. I have done this in one of my codes, so I post the link for simplicity:

In a .pxd file, I define the interface cyfunc_d_d: https://github.com/pdebuyl/skl1/blob/master/skl1/core.pxd
I can re-use this interface in the "base" cython module https://github.com/pdebuyl/skl1/blob/master/skl1/euler.pyx and also in a "user-defined" module.

The final code makes plain "cython-cython" calls while allowing the passing of objects at the Cython level

I adapted the code to your problem:

test_interface.pxd

cdef class cyfunc:                                                                                                                         
    cpdef double f(self, double x)                                                                                                         

cdef class pyfunc(cyfunc):                                                                                                                 
    cdef object py_f                                                                                                                       
    cpdef double f(self, double x)

test_interface.pyx

cdef class cyfunc:
    cpdef double f(self, double x):
        return 0
    def __cinit__(self):
        pass


cdef class pyfunc(cyfunc):
    cpdef double f(self, double x):
        return self.py_f(x)
    def __init__(self, f):
        self.py_f = f

setup.py

from setuptools import setup, Extension                                                                                                    
from Cython.Build import cythonize                                                                                                         

setup(                                                                                                                                     
    ext_modules=cythonize((Extension('test_interface', ["test_interface.pyx"]),                                                            
                          Extension('test_module', ["test_module.pyx"]))                                                                   
                      )                                                                                                                    
)

test_module.pyx

from test_interface cimport cyfunc, pyfunc                                                                                                 

cpdef min_arg(f, int N):                                                                                                                   
    cdef double x = 100000.                                                                                                                
    cdef int best_i = -1                                                                                                                   
    cdef int i                                                                                                                             
    cdef double current_value                                                                                                              

    cdef cyfunc py_f                                                                                                                       

    if isinstance(f, cyfunc):                                                                                                              
        py_f = f                                                                                                                           
        print('cyfunc')                                                                                                                    
    elif callable(f):                                                                                                                      
        py_f = pyfunc(f)                                                                                                                   
        print('no cyfunc')                                                                                                                 
    else:                                                                                                                                  
        raise ValueError("f should be a callable or a cyfunc")                                                                             

    for i in range(N):                                                                                                                     
        current_value = py_f.f(i)                                                                                                          
        if current_value < x:                                                                                                              
            x = current_value                                                                                                              
            best_i = i                                                                                                                     
    return best_i                                                                                                                          

def py_f(x):                                                                                                                               
    return (x-5)**2                                                                                                                        

cdef class cy_f(cyfunc):                                                                                                                   
    cpdef double f(self, double x):                                                                                                        
        return (x-5)**2

To use:

python3 setup.py build_ext --inplace
python3 -c 'import test_module ; print(test_module.min_arg(test_module.cy_f(), 10))'
python3 -c 'import test_module ; print(test_module.min_arg(test_module.py_f, 10))'

Pass cython functions via python interface

1 Answers1

Linked