ACCESS_VIOLATION calling Btrieve BTRCALL function from Rust

Question

I am attempting to call Btrieve (a very old database engine) from Rust. This is a bit long, but this is my first attempt at FFI from Rust and I wanted to describe everything I have done.

The Btrieve engine is implemented in a DLL, w3btrv7.dll, which is a 32-bit DLL. I have made an import library for it using 32-bit MSVC tools (it doesn't come with an official one):

lib /Def:w3btrv7.def /Out:w3btrv7.lib /Machine:x86

I then installed the 32-bit Rust toolchain stable-i686-pc-windows-msvc and set it as my default. Bindgen barfs on the official Btrieve headers so I had to make my own. Luckily we only need to wrap a single function, BTRCALL.

I have this in my wrapper.h:

short int BTRCALL(
    unsigned short  operation,
    void*           posBlock,
    void*           dataBuffer,
    unsigned short* dataLength,
    void*           keyBuffer,
    unsigned char   keyLength,
    char            ckeynum);

I am linking as:

println!("cargo:rustc-link-lib=./src/pervasive/w3btrv7");

Which seems to work: the program runs, is a 32-bit exe, and I can see in Process Explorer that it has loaded w3btrv7.dll.

When I send the header through bindgen I get:

extern "C" {
    pub fn BTRCALL(
        operation: ::std::os::raw::c_ushort,
        posBlock: *mut ::std::os::raw::c_void,
        dataBuffer: *mut ::std::os::raw::c_void,
        dataLength: *mut ::std::os::raw::c_ushort,
        keyBuffer: *mut ::std::os::raw::c_void,
        keyLength: ::std::os::raw::c_uchar,
        ckeynum: ::std::os::raw::c_char,
    ) -> ::std::os::raw::c_short;
}

The types and sizes all seem to tally up correctly, and they match a DllImport I have from a C# application which works perfectly:

[DllImport("w3btrv7.dll", CharSet = CharSet.Ansi)]
private static extern short BTRCALL(
    ushort operation, // In C#, ushort = UInt16.
    [MarshalAs(UnmanagedType.LPArray, SizeConst = 128)] byte[] posBlock,
    [MarshalAs(UnmanagedType.LPArray)] byte[] dataBuffer,
    ref ushort dataLength,
    [MarshalAs(UnmanagedType.LPArray)] byte[] keyBuffer,
    byte keyLength,     // unsigned byte
    char keyNumber);    // 2 byte char

The keyNumber is slightly different, but I have tried both bytes and shorts in both signed and unsigned variations, and it still doesn't work.

Unfortunately when I run my program it blows up after the first call to BTRCALL. (Well, actually it's when the function that this call is in returns). I've extracted all the params into local variables and checked their types and all looks correct:

let op: u16 = 0;
let mut pos_block: [u8; 128] = self.pos_block.clone();
let pos_block_ptr: *mut std::ffi::c_void = pos_block.as_mut_ptr() as *mut _;
let mut data_buffer: [u8; 32768] = self.data_buffer.clone();
let data_buffer_ptr: *mut std::ffi::c_void = data_buffer.as_mut_ptr() as *mut _;
let mut data_length: u16 = data_buffer.len() as u16;
let mut key_buffer: [u8; 256] = self.key_buffer.clone();
let key_buffer_ptr: *mut std::ffi::c_void = key_buffer.as_mut_ptr() as *mut _;
let key_length: u8 = 255; //self.key_length;
let key_number: i8 = self.key_number.try_into().unwrap();

let status: i16 = BTRCALL(
    op,
    pos_block_ptr,
    data_buffer_ptr,
    &mut data_length,
    key_buffer_ptr,
    key_length,
    key_number
);

It crashes the program with

error: process didn't exit successfully: `target\debug\blah.exe` (exit code: 0xc0000005, STATUS_ACCESS_VIOLATION)

From what I have read, this is probably due to an improper address access.

Indeed, when I put some tracing in to check the variables there is some very interesting behaviour, in that my local variables which are passed by value seem to be getting overwritten. The log here just dumps the first 30 bytes of the buffers because the rest is just zeros:

pos_block = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
pos_block_ptr = 0xad6524
data_buffer = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
data_buffer_ptr = 0xad65a8
data_length = 32768
key_buffer = [34, 67, 58, 92, 116, 101, 109, 112, 92, 99, 115, 115, 92, 120, 100, 98, 92, 67, 65, 83, 69, 46, 68, 66, 34, 0, 0, 0, 0, 0]
key_buffer_ptr = 0xade5b0
key_length = 255
key_number = 0

>>>>>>>>>>>>>>> AFTER THE CALL TO BTRCALL:

pos_block = [0, 0, 0, 0, 0, 0, 0, 0, 0, 76, 203, 0, 0, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 0, 0, 0, 0, 0]
pos_block_ptr = 0x0
data_buffer = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
data_buffer_ptr = 0x42442e45
data_length = 0
key_buffer = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
key_buffer_ptr = 0x0
key_length = 173
key_number = 0
BTRCALL() returned B_NO_ERROR

Notice pos_block_ptr has been set to 0, among other things. In contrast, a successful execution of the exact same call from the C# code simply writes some data into the first 18 bytes of pos_block and doesn't change any of the other variables.

It's as if it went a bit berserk and just started overwriting memory...

At this point I don't know what to try next.

It's hard to answer your question because it doesn't include a [MRE]. You've neglected to provide the C and Rust definitions for all of the types used in the function call. You also are using `self` and haven't showm that structure. Please [edit] your question to include this additional info. Ideally, use the [Rust-specific MRE tips](//stackoverflow.com/tags/rust/info) to reduce your original code to just a single `main` function and variable declarations. Thanks! — Shepmaster, Jan 01 '20 at 16:54
I have edited the Rust code to include the types of the parameters to the call. It is not possible to provide a reproducible example because others will not have the DLLs needed. — Philip Daniels, Jan 01 '20 at 17:08
I understand about the DLL (although it seems limiting to assume that *no one* will have it), but that doesn't mean that you shouldn't follow the same process to reduce your local code down as mimimal as possible while producing the same error and then post all of your code. — Shepmaster, Jan 01 '20 at 17:11
Since you are reverse-engineering your DLL, are you sure you've got the right calling convention? — Shepmaster, Jan 01 '20 at 17:17
No I wasn't sure...I wasn't even sure what that was. Just checked over at https://doc.rust-lang.org/nomicon/ffi.htm land changing the declaration to `extern "stdcall" { ...` and it works! — Philip Daniels, Jan 01 '20 at 17:26
What do you mean by "Bindgen barfs on the official Btrieve headers so I had to make my own?" Which Btrieve headers are you using? What version of Btrieve are you using? — mirtheil, Jan 01 '20 at 17:28

score 0 · Accepted Answer · edited Jan 01 '20 at 17:29

Changing the declaration from extern "C" to extern "stdcall" works:

extern "stdcall" {
    pub fn BTRCALL(
        operation: ::std::os::raw::c_ushort,
        posBlock: *mut ::std::os::raw::c_void,
        dataBuffer: *mut ::std::os::raw::c_void,
        dataLength: *mut ::std::os::raw::c_ushort,
        keyBuffer: *mut ::std::os::raw::c_void,
        keyLength: ::std::os::raw::c_uchar,
        ckeynum: ::std::os::raw::c_char,
    ) -> ::std::os::raw::c_short;
}

ACCESS_VIOLATION calling Btrieve BTRCALL function from Rust

1 Answers1