Passing array from Python to C++ using ctypes, unable to process it entirely

Question

I have written the following code to modify my custom python class Point using the ctypes library, following the approch I found in this tutorial. The wrap_function is just a little helper for ease of use, as this is a MWE from a bigger project.

On the python side:

import ctypes
import numpy as np 
libc = ctypes.WinDLL(r'C:\Path\lib.dll')
 
def wrap_function(lib, funcname, restype, argtypes): 
    func = lib.__getattr__(funcname) 
    func.restype = restype 
    func.argtypes = argtypes 
    return func 
 
class Point(ctypes.Structure): 
    _fields_ = [('x', ctypes.c_int), ('xdata', ctypes.c_void_p)]
     
list_of_points = []                                #unused for now 
xdata = np.zeros((40000,), dtype=np.double)
a = Point(1,xdata.ctypes.data) 
b = Point(3,xdata.ctypes.data) 

change_data_for_point = wrap_function(libc,'change_data_for_point', None, [ctypes.POINTER(Point)])
change_data_for_point(a)

And on the C-side:

---header: 

const int N = 40000;
typedef struct {
    double x; 
    double xdata[N];
} Point;

extern "C" LIB_API void change_data_for_point(Point* p);


---source:

void change_data_for_point(Point* p) {
    p->x++; 
    for (int i = 0; i < 40000; i++) {
        p->xdata[i] = 2.0*i;
        if (i % 1000 == 0) printf("xdata at index %d is %f\n", i, p->xdata[i]);
    }
}

When executing the python file in Windows 7 cmd, it prints the following output:

xdata at index 0 is 0.000000
xdata at index 1000 is 2000.000000
 // ... some more ... 
xdata at index 17000 is 34000.000000
xdata at index 18000 is 36000.000000
Traceback (most recent call last):
   File "test.py", line 40, in <module>

Why does it stop at 18.000 ? I tried it several times, sometimes the loop reaches 19 or 20k, but it never gets higher than that. Does it have something to do with the array initialization on the C-side? Did I mess up the parameter passing on the python side?

Bonus question: How can I pass a list of these points to the C-side with ctypes?

Surrounding the critical lines in python with a `try-catch-block` now lets me print `exception: access violation writing 0x0000000006F60000`, so I think I am most likely messing up on the C-side. — masterBroesel, Mar 13 '19 at 07:15
*Point* definition in *C* and *Python* **don't match** (*xdata*). — CristiFati, Mar 13 '19 at 09:42

CristiFati · Accepted Answer · 2019-03-15T13:26:42.047

Although NumPy adds an additional complexity level, every piece of info can be found on [Python 3]: ctypes - A foreign function library for Python.

The (main) problem was that the Point structure was differently defined in C and Python.
Also, the function expects a Point*, so byref must be used (it works without it as well, I don't know whether this is Undefined Behavior's happy case, or ctypes does that silently - due to argtypes).

I've adapted your code in order to work.

dll.c:

#include <stdio.h>

#if defined(_WIN32)
#  define DLL_EXPORT __declspec(dllexport)
#else
#  define DLL_EXPORT
#endif


const int N = 40000;

typedef struct {
    double x;
    double xdata[N];
} Point;


#if defined(__cplusplus)
extern "C" {
#endif

    DLL_EXPORT void change_data_for_point(Point *p);

#if defined(__cplusplus)
}
#endif


void change_data_for_point(Point *p) {
    p->x++; 
    for (int i = 0; i < 40000; i++) {
        p->xdata[i] = 2.0 * i;
        if (i % 10000 == 9999)
            printf("xdata at index %d is %f\n", i, p->xdata[i]);
    }
}

code.py:

#!/usr/bin/env python3

import sys
import ctypes
import numpy as np


DLL_NAME = "./dll.dll"

xdata_dim = 40000  # !!! Must match N (from C) !!!
DoubleArr = ctypes.c_double * xdata_dim

class Point(ctypes.Structure): 
    _fields_ = [
        ("x", ctypes.c_int),
        ("xdata", DoubleArr),
    ]


def wrap_function(lib, funcname, restype, argtypes):
    func = lib.__getattr__(funcname)
    func.restype = restype
    func.argtypes = argtypes
    return func


def main():
    dll = ctypes.CDLL(DLL_NAME)
    #xdata_dim = ctypes.c_int.in_dll(dll, "N")

    xdata = np.zeros((xdata_dim,), dtype=np.double)
    a = Point(1, DoubleArr.from_address(xdata.ctypes.data))
    b = Point(3, DoubleArr.from_address(xdata.ctypes.data))
    change_data_for_point = wrap_function(dll,"change_data_for_point", None, [ctypes.POINTER(Point)])
    change_data_for_point(ctypes.byref(a))
    print(a.xdata[30000])


if __name__ == "__main__":
    print("Python {:s} on {:s}\n".format(sys.version, sys.platform))
    main()

Output:

[cfati@CFATI-5510-0:e:\Work\Dev\StackOverflow\q055124400]> sopr.bat
*** Set shorter prompt to better fit when pasted in StackOverflow (or other) pages ***

[prompt]> "c:\Install\x86\Microsoft\Visual Studio Community\2015\vc\vcvarsall.bat" x64

[prompt]> dir /b
code.py
dll.c

[prompt]> cl /nologo /DDLL /MD /Tp dll.c  /link /NOLOGO /DLL /OUT:dll.dll
dll.c
   Creating library dll.lib and object dll.exp

[prompt]> dir /b
code.py
dll.c
dll.dll
dll.exp
dll.lib
dll.obj

[prompt]> "e:\Work\Dev\VEnvs\py_064_03.06.08_test0\Scripts\python.exe" code.py
Python 3.6.8 (tags/v3.6.8:3c6b436a57, Dec 24 2018, 00:16:47) [MSC v.1916 64 bit (AMD64)] on win32

xdata at index 9999 is 19998.000000
xdata at index 19999 is 39998.000000
xdata at index 29999 is 59998.000000
xdata at index 39999 is 79998.000000
60000.0

@EDIT0:

If you want to handle a list of Points, you can use arrays. Something like:

PointArr = Point * len(list_of_points)
point_arr = PointArr(*list_od_points)

Thanks a lot! That really helped :) How would I go If I wanted to pass a list of points to the C-side to edit them? I already tried nesting the structs on the python side, but upon execution I get an access violation error! — masterBroesel, Mar 15 '19 at 12:51
But you're already *editing* them. The values from the *xdata* array are set from *C*. — CristiFati, Mar 15 '19 at 13:10
Haha, yes I know! Maybe I didnt formulate my intention clear enough - say, I have a list of n Point-objects on the Python-side. Now I want to pass that list to a function (the entire list, not all n points separately - they have to be present on the C side at the same time). This function, on the C-side, should iterate through the list and edit the data from each of these n points. I thought about doing something like `class pointList(ctypes.Structure): _fields_ = [('1',point), ('2',point), ... ('n',point),]` and then passing this to a function, along with the number n. — masterBroesel, Mar 15 '19 at 13:15
Then use an array of points. What you're suggesting is not possible (at least not in that form). Check https://stackoverflow.com/questions/55103298/python-ctypes-read-pointerc-char-in-python/55128773#55128773 - you'd have to do something similar to *CharArr* (somewhere at the end of *main*). — CristiFati, Mar 15 '19 at 13:22
Thanks man! I found a workaround by passing several points separately, but storing them in a `std::vector` on the C-side. Appreciate your help! — masterBroesel, Mar 18 '19 at 12:55
That's right. But using C++ is no problem as long as the objects that are passed to the `ctypes` interface are C only. — masterBroesel, Mar 19 '19 at 13:37

Passing array from Python to C++ using ctypes, unable to process it entirely

1 Answers1