How to convert a ctypes C pointer to a Numpy array?

Question

I'm writing a callback function in Python:

@ctypes.CFUNCTYPE(None, ctypes.c_int, ctypes.c_int, ctypes.c_int, ctypes.c_int, ctypes.c_int, ctypes.c_int, ctypes.c_int, np.ctypeslib.ndpointer(dtype=ctypes.c_ushort))
def FastModeCallback(hCam, x, y, w, h, binx, biny, imageBuffer_ptr):

    image = np.ctypeslib.as_array(imageBuffer_ptr, shape=(h, w))

How can I convert imageBuffer_ptr into a Numpy array of shape=(h,w) and dtype=c.ushort?

I read this answer: Convert Pointer to Array of Unsigned Chars to Numpy Array

and this answer: Getting data from ctypes array into numpy

Both advise using np.ctypeslib.as_array but when I try it I get this error:

Exception ignored on calling ctypes callback function: <function FastModeCallback at 0x00000224E2992DC0>
Traceback (most recent call last):
  File "C:\Users\Dave\data\Code\Python\Cheese\atik.py", line 472, in FastModeCallback
    image = np.ctypeslib.as_array(imageBuffer_ptr, shape=(h, w))
  File "C:\Program_Files\anaconda3\lib\site-packages\numpy\ctypeslib.py", line 521, in as_array
    return array(obj, copy=False)
ValueError: '<P' is not a valid PEP 3118 buffer format string

Google makes it sound like this is a long-standing bug in Python (I'm on 3.8) - what is a workaround?

Maybe https://stackoverflow.com/questions/58049957/numpy-error-when-converting-array-of-ctypes-types-to-void-pointer could help. — CristiFati, Apr 29 '21 at 22:29
Also how are you calling the function? And what arguments are you passing to it? — CristiFati, Apr 29 '21 at 22:39
@CristiFati It's a 3rd party DLL; I don't have the source and don't know how it's called. — nerdfever.com, Apr 30 '21 at 16:30

score 2 · Accepted Answer · answered Apr 30 '21 at 01:37

Listing [Python.Docs]: ctypes - A foreign function library for Python and [NumPy]: C-Types Foreign Function Interface (numpy.ctypeslib). The latter states about the following 2 items (emphasis is mine):

as_array

Create a numpy array from a ctypes array or POINTER.
ndpointer

An ndpointer instance is used to describe an ndarray in restypes and argtypes specifications. This approach is more flexible than using, for example, POINTER(c_double), since several restrictions can be specified, which are verified upon calling the ctypes function.

imageBuffer_ptr is not a "ctypes array or POINTER" (literally: it's not a ctypes.POINTER instance, but ctypes.c_void_p one (there's a subtle difference between the 2)).

The C layer (which in our case lies in the middle) doesn't know about NumPy arrays and so on, so from its PoV, that argument is simply an unsigned short* (doesn't have information about shape, ...). So, on it's other side (in the Python callback), you have to reconstruct the array from the plain C pointer. That can be achieved in 2 ways, like I did in the example below:

callback_ct: Use good-old C way: modify the callback's signature to use ct.POINTER(ct.c_ushort). That is the most straightforward, but the drawback is that it loses all the extra checks when called, meaning that it takes any pointer not just a ndarray one
callback_np: Manually recreate the array from the argument (using a series of conversions). It's more advanced (some might even consider it a bit hackish)

code00.py:

#!/usr/bin/env python

import sys
import ctypes as ct
import numpy as np


Base = ct.c_ushort

NpPtr = np.ctypeslib.ndpointer(
    dtype=Base,
    flags='C_CONTIGUOUS',
    #ndim=1,
)

CtPtr = ct.POINTER(Base)


@ct.CFUNCTYPE(None, ct.c_int, ct.c_int, NpPtr)
def callback_np(w, h, data):
    print("\ncallback_np:")
    #print(data, type(data), type(data).__mro__, hex(data.value))
    #print(data._ndim_, data._dtype_, data._shape_, data._type_, data._objects)
    img = np.ctypeslib.as_array(ct.POINTER(Base).from_address(ct.addressof(data)), shape=(h, w))
    #img = np.ctypeslib.as_array(data, shape=(h, w))  # Original form
    print(img)


@ct.CFUNCTYPE(None, ct.c_int, ct.c_int, CtPtr)
def callback_ct(w, h, data):
    print("\ncallback_ct:")
    img = np.ctypeslib.as_array(data, shape=(h, w))
    print(img)


def main(*argv):
    w = 2
    h = 3
    arr = np.array(range(h * w), dtype=Base)
    print("Array argument: ", arr, type(arr), arr.shape, arr.dtype, arr.ndim)
    callback_ct(w, h, np.ctypeslib.as_ctypes(arr))
    callback_np(w, h, arr)


if __name__ == "__main__":
    print("Python {0:s} {1:d}bit on {2:s}\n".format(" ".join(elem.strip() for elem in sys.version.split("\n")), 64 if sys.maxsize > 0x100000000 else 32, sys.platform))
    rc = main(*sys.argv[1:])
    print("\nDone.")
    sys.exit(rc)

Output:

[cfati@CFATI-5510-0:e:\Work\Dev\StackOverflow\q067323177]> "e:\Work\Dev\VEnvs\py_pc064_03.08.07_test0\Scripts\python.exe" code00.py
Python 3.8.7 (tags/v3.8.7:6503f05, Dec 21 2020, 17:59:51) [MSC v.1928 64 bit (AMD64)] 64bit on win32

Array argument:  [0 1 2 3 4 5] <class 'numpy.ndarray'> (6,) uint16 1

callback_ct:
[[0 1]
 [2 3]
 [4 5]]

callback_np:
[[0 1]
 [2 3]
 [4 5]]

Done.

How to convert a ctypes C pointer to a Numpy array?

1 Answers1

Linked