How do I convert a Rust closure to a C-style callback?

Question

I'm trying to write a Rusty wrapper for a piece of C API. There is one C construct I struggle with:

typedef bool (*listener_t) (int, int);
bool do_it(int x1, int y1, int x2, int y2, listener_t listener)

The function does its job for a range of numbers unless the listener returns false. In that case it aborts computation. I want to have a Rust wrapper like this:

fn do_with_callback<F>(start: (i32, i32), end: (i32, i32), callback: F)
    where F: Fn(i32, i32) -> bool

rust-bindgen created this for me, slightly edited for clarity:

pub type listener_t = Option<extern "C" fn(x: c_int, y: c_int) -> c_bool>;

pub fn TCOD_line(xFrom: c_int, yFrom: c_int,
                 xTo: c_int, yTo: c_int,
                 listener: listener_t) -> c_bool;

How should I convert a closure or a trait reference to a C-style callback in my do_with functions:

pub fn do_with_callback<F>(start: (i32, i32), end: (i32, i32), callback: F) -> Self
    where F: Fn(i32, i32) -> bool
{
    let wrapper = ???;
    unsafe {
        ffi::do_it(start.0, start.1, end.0, end.1, Some(wrapper))
    };
}

Answer 1

You cannot do it unless the C API allows passing a user-provided callback parameter. If it does not, you can only use static functions.

The reason is that closures are not "just" functions. As their name implies, closures "close over" variables from their lexical scope. Each closure has an associated piece of data which holds either values of captured variables (if the move keyword is used) or references to them. This data can be thought of as some unnamed, anonymous struct.

The compiler automatically adds an implementation of the corresponding Fn* traits for these anonymous structs. As you can see, methods on these traits accept self in addition to the closure arguments. In this context, self is the struct on which the trait is implemented. This means that each function which corresponds to a closure also has an additional parameter which contains the closure environment.

If your C API only allows you to pass functions without any user-defined parameters, you cannot write a wrapper which would allow you to use closures. I guess it may be possible to write some global holder for the closures environment, but I doubt it would be easy and safe.

If your C API does allow passing a user-defined argument, then it is possible to do what you want with trait objects:

extern crate libc;

use std::mem;

use libc::{c_int, c_void};

extern "C" {
    fn do_something(f: Option<extern "C" fn(x: c_int, arg: *mut c_void) -> c_int>, arg: *mut c_void) -> c_int;
}

extern "C" fn do_something_handler(x: c_int, arg: *mut c_void) -> c_int {
    let closure: &mut &mut dyn FnMut(i32) -> bool = unsafe { mem::transmute(arg) };
    closure(x as i32) as c_int
}

pub fn do_with_callback<F>(x: i32, mut callback: F) -> bool
    where F: FnMut(i32) -> bool
{
    // reason for double indirection is described below
    let mut cb: &mut dyn FnMut(i32) -> bool = &mut callback;
    let cb = &mut cb;
    unsafe { do_something(Some(do_something_handler), cb as *mut _ as *mut c_void) > 0 }
}

This will only work if do_something does not store the pointer to the callback somewhere. If it does, you need to use a Box<Fn(..) -> ..> trait object and leak it after you pass it to the function. Then, if possible, it should be obtained back from your C library and disposed of. It could look like this:

extern crate libc;

use std::mem;

use libc::{c_int, c_void};

extern "C" {
    fn set_handler(f: Option<extern "C" fn(x: c_int, arg: *mut c_void) -> c_int>, arg: *mut c_void);
    fn invoke_handler(x: c_int) -> c_int;
    fn unset_handler() -> *mut c_void;
}

extern "C" fn do_something_handler(x: c_int, arg: *mut c_void) -> c_int {
    let closure: &mut Box<dyn FnMut(i32) -> bool> = unsafe { mem::transmute(arg) };
    closure(x as i32) as c_int
}

pub fn set_callback<F>(callback: F)
    where F: FnMut(i32) -> bool,
          F: 'static
{
    let cb: Box<Box<dyn FnMut(i32) -> bool>> = Box::new(Box::new(callback));
    unsafe {
        set_handler(Some(do_something_handler), Box::into_raw(cb) as *mut _);
    }
}

pub fn invoke_callback(x: i32) -> bool {
    unsafe { invoke_handler(x as c_int) > 0 }
}

pub fn unset_callback() {
    let ptr = unsafe { unset_handler() };
    // drop the callback
    let _: Box<Box<dyn FnMut(i32) -> bool>> = unsafe { Box::from_raw(ptr as *mut _) };
}

fn main() {
    let mut y = 0;
    set_callback(move |x| {
        y += 1;
        x > y
    });

    println!("First: {}", invoke_callback(2));
    println!("Second: {}", invoke_callback(2));

    unset_callback();
}

Double indirection (i.e. Box<Box<...>>) is necessary because Box<Fn(..) -> ..> is a trait object and therefore a fat pointer, incompatible with *mut c_void because of different size.

Answer 2

The first snippet from Vladimir Matveev no longer works as written. The size of &mut FnMut(i32) -> bool and *mut c_void is different and such casts lead to a crash. Corrected example (playpen):

extern crate libc;

use std::mem::*;

use libc::c_void;

pub fn run<F>(mut callback: F) -> bool
    where F: FnMut(i32) -> bool
{
    let mut cb: &mut FnMut(i32) -> bool = &mut callback;
    println!("sizeof(cb/*-ptr): {}/{}",
             size_of::<*mut FnMut(i32) -> bool>(),
             size_of::<*mut c_void>());

    let ctx = &mut cb as *mut &mut FnMut(i32) -> bool as *mut c_void;
    println!("ctx: {:?}", ctx);
    //----------------------------------------------------------
    // Convert backward
    let cb2: *mut *mut FnMut(i32) -> bool = unsafe { transmute(ctx) };
    println!("cb2: {:?}", cb2);

    // this is more useful, but can't be printed, because not implement Debug
    let closure: &mut &mut FnMut(i32) -> bool = unsafe { transmute(ctx) };

    closure(0xDEAD)
}

fn main() {
    println!("answer: {}",
             run(|x| {
                 println!("What can change nature of a man?");
                 x > 42
             }));
}

Answer 3

In C, a function pointer does not have associated context, which is why usually a C callback function usually carry an extra void* argument pass the context...

typedef bool (*listener_t)(int, int, void* user_data);
bool do_it(void* user_data, int x1, int y1, int x2, int y2, listener_t listener)

... or have an API to let to store the user data...

void api_set_user_data(void* user_data);   // <-- caller set the context
void* api_get_user_data();   // <-- callback use this to retrieve context.

---

If the library you want to wrap does not provide any of the above, you will need to pass the context via other channels, e.g. via a global variable, though that context will be shared across the whole process:

lazy_static! {
    static ref REAL_CALLBACK: Mutex<Option<Box<FnMut(c_int, c_int) -> bool + Send>>> = Default::default();
}

extern "C" fn callback(x: c_int, y: c_int) -> bool {
    if let Some(ref mut real_callback) = *REAL_CALLBACK.lock().unwrap() {
        real_callback(x, y)
    } else {
        panic!("<handle error here>");
    }
}

fn main() {
    *REAL_CALLBACK.lock().unwrap() = Some(Box::new(move |x, y| {
        println!("...");
        true
    }));
    unsafe {
        do_it(callback);
    }
}

It is also possible to create a trampoline function to stick the context directly in the function, but it is extremely difficult and unsafe.

_{Answer manually migrated from https://stackoverflow.com/a/42597209/224671}