0

In comparison with Ctypes, everything being the same except for the syntax, my CFFI implementation quits throwing access violation.

Process finished with exit code -1073741819 (0xC0000005)

Environment: windows 10 (64-bit) Python versions tried:

  • 3.9 (64-bit)
  • 3.10 (64-bit)

I have written two separate implementations of calling a DLL function which returns a point cloud. The DLL accepts a pointer to buffer allocated for a structure and the size of the buffer.

Note: while writing the data to the pointcloud buffer, the DLL takes into consideration only the size passed in iBufSize argument, not for the nPoints variable in the structure.

from cffi import FFI

ffi = FFI()
lib = ffi.dlopen("PointCloud.dll")
ffi.cdef(
"""
typedef struct {
    double x;
    double y;
    double z;
} PointXYZ;

typedef struct {
    uint32_t nPoints;
    PointXYZ * pointcloud;
} PointCloud;

int GetPointCloud(void * ptrPointCloudBuf, int iBufSize);
"""
)

nPts = 5000000
buf_size = nPts * ffi.sizeof('PointXYZ')
pc_struct = ffi.new(
    "PointCloud *", {
        "nPoints": ffi.cast("uint32_t", 0),
        "pointcloud": ffi.new("PointXYZ []", nPts )
    }
)

retcode = lib.GetPointCloud(pc_struct, buf_size)

If the everything is fine, the GetPointCloud() function returns retcode to be SUCCESS with the data written to the pointcloud buffer, else appropriate errorcode to the `retcode'.

The same implementation with Ctypes gets executed without any problem.

from ctypes import *

lib = cdll.LoadLibrary("PointCloud.dll")

class PointXYZ(Structure):
    _fields_ = [
        ('x', c_double),
        ('y', c_double),
        ('z', c_double)
    ]

class PointCloud(Structure):
    _fields_ = [
        ('nPoints', c_uint32),
        ('pointcloud', POINTER(PointXYZ)),
    ]

GetPointCloud = lib['GetPointCloud']
GetPointCloud.restype = c_int
GetPointCloud.argtypes = [POINTER(PointCloud), c_int]

nPts = 5000000
buf_size = nPts * sizeof(PointXYZ)
pc_struct = pointer(
            PointCloud(
                nPoints=c_uint32(0),
                pointcloud=cast((PointXYZ * nPts)(), POINTER(PointXYZ))
            )
        )

retcode = lib.GetPointCloud(pc_struct, buf_size)

Note:

- DLL (pure C implementation) Source code is not availabele to me.

- Both, CFFI and Ctypes, implementations were working smoothly for previous versions of DLL, until they changed something in the DLL, while keeping the external interface same.

The contents of the header file supplied with the DLL. Of course it has a few of other functions. But I have pasted only the part of our interest.

#ifndef POINTCLOUD_H
#define POINTCLOUD_H
#if defined(_MSC_VER)
    #define POINTCLOUD_DLL_EXPORT __declspec(dllexport)        
#else
    #define POINTCLOUD_DLL_EXPORT __attribute__ ((visibility ("default")))
    #define _stdcall
#endif

// Point structs
typedef struct {
    double x;
    double y;
    double z;
} PointXYZ;

struct PointCloud {
    uint32_t number_of_points;
    PointXYZ * pointcloud;
    PointCloud() {
        pointcloud= nullptr;
        number_of_points = 0;
    }
};

extern "C" POINTCLOUD_DLL_EXPORT int  GetPointCloud(void* buffer, int buffer_size);

#endif // POINTCLOUD_H

I have access to the source code of their C/C++ demo application which uses the DLL, which runs absolutely fine. A code snippet for the respective C code is:

struct PointXYZ
{
    double x;
    double y;
    double z;
};
struct PointCloud {
    uint32_t nPoints;
    PointXYZ * pointcloud;
    PointCloud() {
        pointcloud = nullptr;
        nPoints = 0;
    }
};

PointCloud pcloud;
int nPts = 5000000;

pcloud.pointcloud = new PointXYZ[nPts];
buf_size = sizeof(PointXYZ) * nPts;
int n = GetPointCloud(&pcloud, buf_size);   
if(n == 0)
{
    printf("Success, result = %d\n", n);    
}
else
{
    printf("Failure, result = %d\n", n);
}
/*
Do the rest with the point cloud.
*/
Bawoo
  • 1
  • 4
  • Out of curiosity: how many points does the *.dll* function return? – CristiFati Apr 19 '23 at 20:59
  • 1.25 Million, 3 Million, 5 Million or 12 Million. In case of 12 Million, I have to create that big buffer and pass the size in the `buf_size` argument, while calling the GetPointCloud() function. X, Y and Z coordinates are of `double` precision. *Note: It doesn't really matter, if my buffer is bigger but the dll has less data to write. It writes in the `nPoints` variable, how many points it wrote in the buffer. So I know how many points to read from the buffer. The `dll` returns appropriate errorcode in the `retcode` in case the buffer is smaller.* – Bawoo Apr 20 '23 at 07:18
  • Can you try with bigger buffer size? Do you have the *GetPointCloud* *API* description? Or the *.dll* iyself? – CristiFati Apr 20 '23 at 07:19
  • In both implementations (CFFI or Ctypes), I always create biggest possible buffer (12 Million), even if I know that the DLL is going to write only 1.25 Million points. And upon `success`, I check the `nPoints` variable in the structure, to read the number of points. From the DLL's documentation, the API description is this, `int GetPointCloud(void * buffer, int buffer_size);` For the reason I am working in professional environment, I cannot share the DLL. Sincere apologies!! – Bawoo Apr 20 '23 at 07:30
  • No, I figured the function signature out from your code. I meant some documentation of what each parameter is. Is it somewhere publicly available (or maybe I could also download the software (including the *.dll*))? What happens if you try calling it directly from *C* (same parameters)? One curiosity: if the buffer is too small how it behaves? It just throws an error that the buffer is to small (maybe indicating the required size), or fills it out and in the next call(s) fills the rest of the points? – CristiFati Apr 20 '23 at 07:36
  • @CristiFati, since my comment was getting too long, I have mentioned it below. – Bawoo Apr 20 '23 at 08:42
  • If the buffer created was too small but the size passed in the `buf_size` argument was big enough, program quits due to memory access violation. DLL has no mechanism to check actual size of buffer, as it needs only the pointer to it. It compares the size its data with that of passed in `buf_size` argument, and attempts to write the data to the buffer pointed by the pointer. – Bawoo Apr 20 '23 at 08:55
  • @CristiFati, unfortunately it's not that easy. The point cloud actually comes from a hardware. And the DLL communicates with the firmware of the hardware. For you to download the dll and try it out on your end, you will also need the hardware which generates point clouds, sends to the DLL and DLL delivers it to application program. My program being a bit complex (meaning the `buf_size` being in actual calculated from some parameters received from hardware), I have made it simple by hard-coding, because rest of the things are working fine, except for the GetPointCloud() function call. – Bawoo Apr 20 '23 at 09:28
  • I see (kind of expecting it). I was asking about the function documentation to see if I could create a dummy one on my side to reproduce the problem. Didn't something change (parameter types, structure definitions) between the 2 versions? Just to be clear: the 2nd argument specifies the *pointcloud* buffer size in *PointXYZ* structures? – CristiFati Apr 20 '23 at 09:32
  • The function documentation mentions only the signature, while their demo application (C/C++) uses the code I mentioned below in the answers. And yes, the 2nd argument specifies the pointcloud buffer size in PointXYZ structures. – Bawoo Apr 20 '23 at 10:03
  • @CristiFati, sorry one question remained un answered. Between the 2 DLL versions, yes, one thing changed: Old version was accepting a third parameter for number of points as pointer to an integer. In the new version the number of points moved to the structure and has been made as integer instead of pointer to an integer. And because while calling the function, a pointer to the buffer allocated for the structure is passed, it should work. At least when it is working with Ctypes, it SHOULD work with CFFI as well. – Bawoo Apr 20 '23 at 11:09

1 Answers1

0

I've got it, it's an invalid temporary reference resulting in a dangling pointer usage which is Undefined Behavior (common pitfall, especially for Python programmers that come in contact with C).

According to [ReadTheDocs]: Using the ffi/lib objects - Working with pointers, structures and arrays (emphasis is mine):

However, this is always wrong (usage of freed memory):

p = ffi.new("char **", ffi.new("char[]", "hello, world"))
# WRONG!  as soon as p is built, the inner ffi.new() gets freed!

I prepared a small example.

  • dll00.c:

    #include <inttypes.h>
    #include <stdio.h>
    #include <stdlib.h>
    #include <time.h>
    
    #if defined(_WIN32)
    #  define DLL00_EXPORT_API __declspec(dllexport)
    #else
    #  define DLL00_EXPORT_API
    #endif
    
    
    #if defined(__cplusplus)
    extern "C" {
    #endif
    
    DLL00_EXPORT_API int GetPointCloud(void *pBuf, int bSize);
    
    #if defined(__cplusplus)
    }
    #endif
    
    
    typedef struct {
        double x;
        double y;
        double z;
    } PointXYZ;
    
    typedef struct {
        uint32_t nPoints;
        PointXYZ *pointcloud;
    } PointCloud;
    
    
    int GetPointCloud(void *pBuf, int bSize)
    {
        if ((pBuf == NULL) || (bSize <= 0)) {
            return -1;
        }
        unsigned int seed = (unsigned int)time(NULL);
        //printf("%d %d\n", RAND_MAX, seed);
        srand(seed);
        int r = rand();
        printf("From C: Attempting to return %d points\n", r);
        size_t req = r * sizeof(PointXYZ);
        if (req > bSize) {
            return req;
        }
        PointCloud *ppc = (PointCloud*)pBuf;
        for (int i = 0; i < r; ++i)
            ppc->pointcloud[i] = (PointXYZ){i * 3, i * 3 + 1, i * 3 + 2};
        ppc->nPoints = r;
        return 0;
    }
    
  • code00.py:

    #!/usr/bin/env python
    
    import ctypes as cts
    #import random
    import sys
    
    
    DLL_NAME = "./PointCloud.{:s}".format("dll" if sys.platform[:3].lower() == "win" else "so")
    
    class PointXYZ(cts.Structure):
        _fields_ = (
            ("x", cts.c_double),
            ("y", cts.c_double),
            ("z", cts.c_double),
        )
    
    PPointXYZ = cts.POINTER(PointXYZ)
    
    
    class PointCloud(cts.Structure):
        _fields_ = (
            ("nPoints", cts.c_uint32),
            ("pointcloud", PPointXYZ),
        )
    
    PPointCloud = cts.POINTER(PointCloud)
    
    
    def call_dll_func(func, pc, size, pc_transform=lambda arg: arg):
        res = func(pc_transform(pc), size)
        print("\nFunction {:} returned: {:d}".format(func, res))
        if res != 0:
            return
        idxs = (0, pc.nPoints // 2, pc.nPoints - 1)
        print("\nPoints: {:d}\nData ({:d} points):".format(pc.nPoints, len(idxs)))
        for i in idxs:
            p = pc.pointcloud[i]
            print("  {:d}: {:.1f} {:.1f} {:.1f}".format(i, p.x, p.y, p.z))
    
    
    def main(*argv):
        #random.seed()
        dll = cts.CDLL(DLL_NAME)
        GetPointCloud = dll.GetPointCloud
        GetPointCloud.argtypes = ()
        GetPointCloud.restype = cts.c_int
    
        pts = 5000000
        buf_size = pts * cts.sizeof(PointXYZ)
        pc = PointCloud(nPoints=cts.c_uint32(0), pointcloud=cts.cast((PointXYZ * pts)(), PPointXYZ))
    
        call_dll_func(GetPointCloud, pc, buf_size, pc_transform=lambda arg: cts.byref(arg))
    
    
    if __name__ == "__main__":
        print("Python {:s} {:03d}bit on {:s}\n".format(" ".join(elem.strip() for elem in sys.version.split("\n")),
                                                       64 if sys.maxsize > 0x100000000 else 32, sys.platform))
        rc = main(*sys.argv[1:])
        print("\nDone.\n")
        sys.exit(rc)
    
  • code01.py:

    #!/usr/bin/env python
    
    import sys
    
    from cffi import FFI
    
    from code00 import DLL_NAME, \
        call_dll_func
    
    
    ffi = FFI()
    
    ffi.cdef(
    """
    typedef struct {
        double x;
        double y;
        double z;
    } PointXYZ;
    
    typedef struct {
        uint32_t nPoints;
        PointXYZ * pointcloud;
    } PointCloud;
    
    int GetPointCloud(void *pBuf, int bSize);
    """
    )
    
    
    def main(*argv):
        dll = ffi.dlopen(DLL_NAME)
    
        pts = 5000000
        buf_size = pts * ffi.sizeof("PointXYZ")
        pc_buf = ffi.new("PointXYZ []", pts)  # @TODO - cfati: - Take it outside
        pc = ffi.new(
            "PointCloud *", {
                "nPoints": ffi.cast("uint32_t", 0),
                "pointcloud": pc_buf,
            }
        )
    
        GetPointCloud = dll.GetPointCloud
    
        call_dll_func(GetPointCloud, pc, buf_size)
    
    
    if __name__ == "__main__":
        print("Python {:s} {:03d}bit on {:s}\n".format(" ".join(elem.strip() for elem in sys.version.split("\n")),
                                                       64 if sys.maxsize > 0x100000000 else 32, sys.platform))
        rc = main(*sys.argv[1:])
        print("\nDone.\n")
        sys.exit(rc)
    

Output:

[cfati@CFATI-5510-0:e:\Work\Dev\StackExchange\StackOverflow\q076053999]> sopr.bat
### Set shorter prompt to better fit when pasted in StackOverflow (or other) pages ###

[prompt]> "c:\Install\pc032\Microsoft\VisualStudioCommunity\2019\VC\Auxiliary\Build\vcvarsall.bat" x64 > nul

[prompt]>
[prompt]> dir /b
code00.py
code01.py
dll00.c

[prompt]>
[prompt]> cl /nologo /MD /DDLL dll00.c  /link /NOLOGO /DLL /OUT:PointCloud.dll
dll00.c
   Creating library PointCloud.lib and object PointCloud.exp

[prompt]>
[prompt]> dir /b
code00.py
code01.py
dll00.c
dll00.obj
PointCloud.dll
PointCloud.exp
PointCloud.lib

[prompt]>
[prompt]> :: CTypes
[prompt]> "e:\Work\Dev\VEnvs\py_pc064_03.10_test0\Scripts\python.exe" ./code00.py
Python 3.10.9 (tags/v3.10.9:1dd9be6, Dec  6 2022, 20:01:21) [MSC v.1934 64 bit (AMD64)] 064bit on win32

From C: Attempting to return 9724 points

Function <_FuncPtr object at 0x0000020CA1C40EE0> returned: 0

Points: 9724
Data (3 points):
  0: 0.0 1.0 2.0
  4862: 14586.0 14587.0 14588.0
  9723: 29169.0 29170.0 29171.0

Done.


[prompt]>
[prompt]> :: CFFI
[prompt]> "e:\Work\Dev\VEnvs\py_pc064_03.10_test0\Scripts\python.exe" ./code01.py
Python 3.10.9 (tags/v3.10.9:1dd9be6, Dec  6 2022, 20:01:21) [MSC v.1934 64 bit (AMD64)] 064bit on win32

From C: Attempting to return 9740 points

Function <cdata 'int(*)(void *, int)' 0x00007FFB33351020> returned: 0

Points: 9740
Data (3 points):
  0: 0.0 1.0 2.0
  4870: 14610.0 14611.0 14612.0
  9739: 29217.0 29218.0 29219.0

Done.

Might also want ro check [SO]: C function called from Python via ctypes returns incorrect value (@CristiFati's answer) for a(nother) pitfall: this time when working with CTypes (calling functions).

CristiFati
  • 38,250
  • 9
  • 50
  • 87
  • thank you very much for your valuable answer. However, taking it out did not work for me. With CFFI it still gives me the same problem. In fact, before they moved the `nPoints` inside the structure, everything worked fine. I used to allocate the memory `"pointcloud": ffi.new("PointXYZ []", pts)` inside the structure only. The pointcloud received was as expected and I could visualize it correctly using e.g. Open3D. And for those older versions of the DLL, the signature was, `int GetPointCloud(void *pBuf, int bSize, int * nPoints);`. Ctypes works irrespective of inside or outside. – Bawoo Apr 20 '23 at 19:55
  • To be on safer side, I have taken it out and am going to keep it like that. – Bawoo Apr 20 '23 at 20:01
  • It's ***U**ndefined **B**ehavior* either way. The fact sometimes works and other times crashes is a matter of hazard. So, this kind of answers the current problem (exposed so far). If things don't work in your environment, there is some "hidden" stuff. The *.dll* and calling code must be in sync. If you still have issues, they aren't. As I see you switching between *.dll*s and structure definitions, I tend to think you are mismatching the 2. By "working" I assume is "not crashing". But in case of mismatches (some) data will be corrupt. – CristiFati Apr 20 '23 at 20:07
  • Also, the *.dll* should have come with one header file (or more), which should contain the correct structure definitions, corresponding to the function signature. Or the vendor should provide some form of documentation. – CristiFati Apr 20 '23 at 20:10
  • please check the answer below. I have pasted the header file contents of interest. – Bawoo Apr 21 '23 at 11:20
  • @Bawoo: thank you. But as a side note, that information belongs to the question (edit it, not in an answer). Same for the other. – CristiFati Apr 21 '23 at 13:18
  • Also what *Python* version (and bitness) are you using? – CristiFati Apr 21 '23 at 13:24
  • updated the question and mentioned my environment details also in the question. Thanks. – Bawoo Apr 21 '23 at 13:31
  • I had a chance to look into what is happening in the `C` implementation side. The implementation is much more complex. What I call from the DLL is a wrapper (mapped) function, which inturn calls another GetPointCloud() function and the pointer to the pointcloud structure is handled by several other function calls. With my 0.1% knowledge in `C(++)`, I can hardly understand anything happening there. Thanks a ton again for your efforts for solving my problem. – Bawoo Apr 26 '23 at 15:25