How do I use the Rust memory allocator for a C library that can be provided an allocator?

Question

I'm writing Rust bindings to a C library which has the option to use a third-party memory allocator. Its interface looks like this:

struct allocator {
    void*(*alloc)(void *old, uint);
    void(*free)(void*);
};

The corresponding Rust struct is, I guess, the following:

#[repr(C)]
#[derive(Copy, Clone, Debug, PartialEq)]
pub struct Allocator {
    alloc: Option<extern "C" fn(*mut c_void, c_uint) -> *mut c_void>,
    free: Option<extern "C" fn(*mut c_void)>,
}

How can I implement these two extern functions that should mimic the allocator? I did not find anything really looking like the allocator API in Rust (I understand why however), so I'm curious if it is possible.

https://www.reddit.com/r/rust/comments/2eqdg2/allocate_a_vec_on_cs_heap/ There are some thoughts on a similar topic, although the allocations lifetime management may be tricky. — snuk182, Aug 31 '16 at 13:21
As you point out, that topic is about making Rust use the same allocator as C. That's possible with [custom allocators](https://doc.rust-lang.org/stable/book/custom-allocators.html). — Shepmaster, Aug 31 '16 at 13:53

Shepmaster · Accepted Answer · 2016-08-31T16:58:03.533

It's not as easy as you might like.

The allocation methods are exposed in the heap module of the alloc crate.

Creating some wrapper methods and populating the struct is straight-forward, but we quickly run into an issue:

#![feature(heap_api)]

extern crate libc;
extern crate alloc;

use libc::{c_void, c_uint};
use alloc::heap;

#[repr(C)]
#[derive(Copy, Clone, Debug, PartialEq)]
pub struct Allocator {
    alloc: Option<extern "C" fn(*mut c_void, c_uint) -> *mut c_void>,
    free: Option<extern "C" fn(*mut c_void)>,
}


extern "C" fn alloc_ext(old: *mut c_void, size: c_uint) -> *mut c_void {
    if old.is_null() {
        heap::allocate(size as usize, align) as *mut c_void
    } else {
        heap::reallocate(old as *mut u8, old_size, size as usize, align) as *mut c_void
    }
}

extern "C" fn free_ext(old: *mut c_void) {
    heap::deallocate(old as *mut u8, old_size, align);
}

fn main() {
    Allocator {
        alloc: Some(alloc_ext),
        free: Some(free_ext),
    };
}

The Rust allocator expects to be told the size of any previous allocation as well as the desired alignment. The API you are matching doesn't have any way of passing that along.

Alignment should (I'm not an expert) be OK to hardcode at some value, say 16 bytes. The size is trickier. You will likely need to steal some old C tricks and allocate a little bit extra space to store the size in. You can then store the size and return a pointer just past that.

A completely untested example:

#![feature(alloc, heap_api)]

extern crate libc;
extern crate alloc;

use libc::{c_void, c_uint};
use alloc::heap;
use std::{mem, ptr};

#[repr(C)]
#[derive(Copy, Clone, Debug, PartialEq)]
pub struct Allocator {
    alloc: Option<extern "C" fn(*mut c_void, c_uint) -> *mut c_void>,
    free: Option<extern "C" fn(*mut c_void)>,
}

const ALIGNMENT: usize = 16;

extern "C" fn alloc_ext(old: *mut c_void, size: c_uint) -> *mut c_void {
    unsafe {
        // Should check for integer overflow
        let size_size = mem::size_of::<usize>();
        let size = size as usize + size_size;

        let memory = if old.is_null() {
            heap::allocate(size, ALIGNMENT)
        } else {
            let old = old as *mut u8;
            let old = old.offset(-(size_size as isize));
            let old_size = *(old as *const usize);
            heap::reallocate(old, old_size, size, ALIGNMENT)
        };

        *(memory as *mut usize) = size;
        memory.offset(size_size as isize) as *mut c_void
    }
}

extern "C" fn free_ext(old: *mut c_void) {
    if old.is_null() { return }

    unsafe {
        let size_size = mem::size_of::<usize>();

        let old = old as *mut u8;
        let old = old.offset(-(size_size as isize));
        let old_size = *(old as *const usize);

        heap::deallocate(old as *mut u8, old_size, ALIGNMENT);
    }
}

fn main() {
    Allocator {
        alloc: Some(alloc_ext),
        free: Some(free_ext),
    };

    let pointer = alloc_ext(ptr::null_mut(), 54);
    let pointer = alloc_ext(pointer, 105);
    free_ext(pointer);
}

Isn't [... using Vec as an allocator ...] the more high-level solution?

That's certainly possible, but I'm not completely sure how it would work with reallocation. You'd also have to keep track of the size and capacity of the Vec in order to reconstitute it to reallocate / drop it.

You could in principle also store the allocation sizes in a `HashMap` or similar. I'm not sure you'd gain anything over overallocating and storing in the allocation, though (except for a better chance of detecting a bad `free` from the C side). — Chris Emerson, Aug 31 '16 at 13:40
@ChrisEmerson good point! However, when I've tried similar things in the past, I've experienced abnormally poor performance. Without knowing exactly why, I think it has to do with poor cache locality. Multiple threads is also more painful with a shared collection. — Shepmaster, Aug 31 '16 at 13:51
UPD: Here is where I faced the C's unsafety. The called library code was using my **free_ext** implementation with a null pointer, so I had to add its check explicitly: `if old.is_null() { println!("no dealloc for empty pointer"); return; } ` — snuk182, Sep 04 '16 at 14:22
Btw, [Vec as allocator](http://github.com/rust-lang/rust/issues/27700#issuecomment-16901471%E2%80%8C%E2%80%8B3) approach also works. May be useful to the ones who don't want or cannot use unstable API. — snuk182, Sep 04 '16 at 19:00
Also, as [new allocator API](https://github.com/rust-lang/rust/issues/32838) has been introduced, the answer is also [modified a bit](https://github.com/snuk182/nuklear-rust/commit/ca24f6ab463650f092ad98e5c8d7ae142d5dd8fd). — snuk182, Sep 28 '17 at 15:18

How do I use the Rust memory allocator for a C library that can be provided an allocator?

1 Answers1

Linked