Efficient conversion from C array of struct to NumPy array with ctypes

Question

Following this question where I was trying to use a C++ DLL with Cython, with a tutorial fit to my case that never worked, I decided to use ctypes. I now successfully call the function that interests me in my DLL using ctypes thanks to heavy SO browsing. I am now faced with the problem of using the results that is an array of struct in Python.

This C function is as follows:

void myfun(
                 double         a,
                 //...more double parameters
                 int            max_iter,
                 int *          nb_iter,
                 myStruct *     res_arr,
                 bool *         ok
                );

with myStruct defined as follows:

typedef struct  {
             double dat;

             int    k;
             int    m;
            // ... more int

             double b;
             double v;
             //...more double

            } myStruct;

I call this function through the following Python code:

import ctypes
lib = ctypes.CDLL('PATH_TO_DLL\\lib.dll')

myFunPy = getattr(lib,"?myFun@@YANNNNN_BUNCH_OF_Ns_NNNHPEAHPEAUmyStruct@@PEA_N@Z") # name found through dumpbin.exe (due to C++)

class myStruct(ctypes.Structure):
    _fields_ = [("k", ctypes.c_int),
                ("m", ctypes.c_int),
                #...more int parameters

                ("b", ctypes.c_double),
                ("v", ctypes.c_double)
                #...more double parameters
               ]

myFunPy.argtypes = [ctypes.c_double,
                   // ... more double parameters

                   ctypes.c_int,
                   ctypes.POINTER(ctypes.c_int),
                   ctypes.POINTER(myStruct),
                   ctypes.POINTER(ctypes.c_bool)]

myFunPy.restype = ctypes.c_void_p

max_iter = 10000
a = ctypes.c_double(0.1)
// ... more double parameters definitions

nb_iter = ctypes.c_int(0) # value doesn't matter, it is initialized in myFun
ok = ctypes.c_bool(True)

res_arr = (myStruct * max_iter)()

myFunPy(a, ..., max_iter, ctypes.byref(nb_iter), res_arr, ctypes.byref(ok))

Now myFun modifies res_arr which is an array of struct as can be seen from the above code. It is exactly

<__main__.myStruct_Array_10000 at 0x97966c8>)

after the code shown above, but I cannot understand how to convert it to a NumPy array for future use efficiently.

Sure, I could do for loops with stuff like for field, _ in struct._fields_ as shown here, but that is not the point since I use the DLL to make my computations faster (I really saw the difference in the execution time). res_arr ranges from 200 kb to 1 Mb and has tens of thousands of lines and some dozens of columns, so I'm sure that there is a way to not go through all of it with loops, but I can't figure out how to do so nicely.

It seems that if it wasn't an array of struct, it would be easier. There are a few SO questions (also here, here, here, and here) close to this subject but it's either about converting just a struct, just an array, or something close but never exactly like me, and I wasn't successful in adapting these solutions, so maybe there's a way to base an answer on that, but in any case I'm all ears.

Answer 1

We have almost the same problem but in my case, I was using CUDA DLL so my compiler was nvcc. But I believe that this can also be done by the normal g++ compiler. In any case, here are the steps I did in order to convert array of structs from my CPP file to useable Python list/array. I will not go through your code; instead, I will just give you example which can be be found here: https://github.com/jcbacong/python-cpp.git

But the important steps are summarized as follows:

1. Create a .cpp file with the necessary header file containing the extern "C" declaration for the function. In my .cpp file, I returned an array of struct instead of returning a void.

2. Create a .dll file using your compiler. Again, in my case, it was nvcc. The sample code I linked through my github account was compiled using nvcc.

3. In your .py file:

   3.1 Create a Python class with ctypes.Structure in order to replicate the struct definition in your .cpp/.h file.

   3.2 Initialize your input/output using argtype/restype. Since my .cpp function is returning an array of struct, the restype is given by ctypes.Pointer(<your Python Class(ctypes.Structure)>).

   3.3 I converted all inputs to readable ctypes. After calling the function in my .py file, the resulting array of structs (_results in the example) can be converted to Python list using (results = _results[:ARRAY_SIZE]).

I hope this helps!!