Dereference void pointer to numpy array or list

Question

I've received a void pointer from a foreign function via ctypes, containing an array of c_double arrays: [[12.0, 13.0], [14.0, 15.0], …]

I'm accessing it via the restype parameter:

from ctypes import Structure, POINTER, c_void_p, c_size_t, c_double, c_uint32, c_char_p, cast, cdll


class _CoordResult(Structure):
    """ Container for returned FFI coordinate data """
    _fields_ = [("coords", _FFIArray)]

class _FFIArray(Structure):
    """
    Convert sequence of float lists to a C-compatible void array
    example: [[1.0, 2.0], [3.0, 4.0]]

    """
    _fields_ = [("data", c_void_p),
                ("len", c_size_t)]

def _void_array_to_nested_list(res, _func, _args):
    """ Dereference the FFI result to a list of coordinates """
    shape = (res.coords.len, 2)
    array_size = np.prod(shape)
    mem_size = 8 * array_size
    array_str = string_at(res.coords.data, mem_size)
    array = [list(pair) for pair in ((POINTER(c_double * 2).from_buffer_copy(res.coords)[:res.coords.len]))]
    drop_array(res.coords)
    return array

decode_polyline = lib.decode_polyline_ffi
decode_polyline.argtypes = (c_char_p, c_uint32)
decode_polyline.restype = _CoordResult
decode_polyline.errcheck = _void_array_to_nested_list

However, this gives me back nonsense values, because the pointer dereference in _void_array_to_nested_list is wrong.

The solution doesn't have to use NumPy, but that seems like the best approach.

A [mcve] would be useful. Have you verified somehow that the function being called returns sensible values in the first place i.e. tested the code without ctypes? — J.J. Hakala, Jul 19 '16 at 09:18
@J.J.Hakala Done! Yes, the function returns sensible values when tested without ctypes. — urschrei, Jul 19 '16 at 09:32

score 2 · Accepted Answer · answered Jul 18 '16 at 18:09

2

I can't test this right now, but this is what I would try:

import numpy as np

result = ...
shape = (10, 2)
array_size = np.prod(shape)
mem_size = 8 * array_size
array_str = ctypes.string_at(result, mem_size)
array = np.frombuffer(array_str, float, array_size).reshape(shape)

array will be read only, copy it if you need a writable array.

answered Jul 18 '16 at 18:09

Bi Rico

25,283
3
52
75

Is there any reason this should be failing on Windows? It works perfectly on *nix and OSX, using `dtype="float64"`, but returns nonsense values on 32-bit and 64-bit Windows about 50% of the time. – urschrei Jul 26 '16 at 11:33
1

It's hard to debug without more info, but this sounds like it may be related to the windows long issue. Check your pointer and make sure you're using a pointer type appropriate for the platform. http://stackoverflow.com/questions/22513445/python-handles-long-ints-differently-on-windows-and-unix – Bi Rico Jul 26 '16 at 15:36

J.J. Hakala · Answer 2 · 2016-07-27T13:19:40.327

Here is a solution that uses ctypes.cast or numpy.ctypeslib.as_array, and no ctypes.string_at just in case if it makes an extra copy of memory region.

class _FFIArray(Structure):
    _fields_ = [("data", c_void_p), ("len", c_size_t)]

class Coordinate(Structure):
    _fields_ = [("latitude", c_double), ("longitude", c_double)]

class Coordinates(Structure):
    _fields_ = [("data", POINTER(Coordinate)), ("len", c_size_t)]

decode_polyline = lib.decode_polyline_ffi
decode_polyline.argtypes = (c_char_p, c_uint32)
decode_polyline.restype = _FFIArray

# assuming that the second argument is the length of polyline, 
# although it should not be needed for `\0` terminated string
res = decode_polyline(polyline, len(polyline))

nres = Coordinates(cast(res.data, POINTER(Coordinate)), res.len)
for i in range(nres.len):
    print(nres.data[i].latitude, nres.data[i].longitude)

# if just a numpy (np) array is needed
xs = np.ctypeslib.as_array((c_double * res.len * 2).from_address(res.data))
# "New view of array with the same data."
xs = xs.view(dtype=[('a', np.float64), ('b', np.float64)], type=np.ndarray)
xs.shape = res.len

Dereference void pointer to numpy array or list

2 Answers2