I make heavy use of numerical analysis libraries in Python such as Scipy's optimization and integration routines, and many of these routines take a callback which evaluates some sort of objective function as their main argument. For performance reasons I frequently implement these callbacks in native code (typically using Cython or Numba which generate C code that is compiled, linked to, and then encapsulated in a Python object). Many Scipy routines now accept a LowLevelCallable instance, which is such an object that contains a pointer to a C function and information on its signature. If the routine itself is implemented in C or Cython this can greatly reduce overhead by bypassing the Python interpreter entirely.
What I am trying to figure out the best way to do is to create Python functions which take these types of native callback objects, transform them in some way, and return another native callback. An example would be a function that takes a callback that predicts y given x along with an array of observed y values, and returns another callback that computes the error in predictions compared to observations. The result could then be passed to scipy.optimize.minimize().
An implementation for regular Python functions would be something like this:
import numpy as np
def make_error_callback(f, y_obs):
def g(x):
y = f(x)
return np.sum((y - y_obs) ** 2)
return g
The goal would be that if f is a native callback that can be evaluated without invoking the Python interpreter, then g should be as well. make_error_callback would probably have to be implemented in Cython or something similar (at worst actual C using Python's C API). The actual Python types of f and g would be something that encapsulates a C function pointer, like scipy.LowLevelCallable.
