Python ctypes cdll.LoadLibrary, instantiate an object, execute its method, private variable address truncated

Question

I wrote a dll library in c, compile with vs2017 64-bit, and try to load it with python3.6 64-bit. However the object's member variable's address got truncated to 32-bit.

Here's my sim.c file, which is compiled to sim.dll:

class Detector {
public:
    Detector();
    void process(int* pin, int* pout, int n);

private:
    int member_var;
};

Detector::Detector()
{
    memset(&member_var, 0, sizeof(member_var));
    myfile.open("addr_debug.txt");
    myfile << "member_var init address: " << &member_var << endl;
}
void Detector::process(int* pin, int* pout, int n);
{
    myfile << "member_var process address: " << &member_var << endl;
    myfile.close();
}

#define DllExport   __declspec( dllexport )  

extern "C" {
    DllExport Detector* Detector_new() { return new Detector(); }
    DllExport void Detector_process(Detector* det, int* pin, int* pout, int n)
    {
        det->process(pin, pout, n);
    }
}

Here's my python script:

from ctypes import cdll
lib = cdll.LoadLibrary(r'sim.dll')

class Detector(object):
    def __init__(self):
        self.obj = lib.Detector_new()

    def process(self,pin, pout, n):
        lib.Detector_process(self.obj,pin, pout, n)

detector = Detector()

n = 1024
a = np.arange(n, dtype=np.uint32)
b = np.zeros(n, dtype=np.int32)

aptr = a.ctypes.data_as(ctypes.POINTER(ctypes.c_int))
bptr = b.ctypes.data_as(ctypes.POINTER(ctypes.c_int))

detector.process(aptr, bptr, n)

Here's the address of the member_var in addr_debug.txt:

member_var init address:    0000025259E123C4
member_var process address: 0000000059E123C4

So accessing it trigger memory access error:

OSError: exception: access violation reading 0000000059E123C4

---

Some attempts I tried to understand the issue:

• Define member_var as public instead of private, not help, address still truncated.
• Define member_var as global variable, then the address is ok. So I guess the member_var address truncation happens either when returning the object to python, or passing the object back to dll.

Answer 1

Always (CORRECTLY) specify argtypes and restype for functions defined in C, otherwise (C89 style) they will default to int (generally 32bit), generating !!! Undefined Behavior !!!. On 64bit, addresses (larger than 2 GiB) will be truncated (which is exactly what you're experiencing). Check [SO]: C function called from Python via ctypes returns incorrect value (@CristiFati's answer) for more details.

Also, when running into issues, don't forget about [Python.Docs]: ctypes - A foreign function library for Python.

Below it's an adapted version of your code.

detector.cpp:

#include <stdio.h>
#include <memory.h>

#include <fstream>

#define SIM_EXPORT __declspec(dllexport)


#define C_TAG "From C"
#define PRINT_MSG_3SPI(ARG0, ARG1, ARG2) printf("%s - [%s] (%d) - [%s]:  %s: 0x%0p(%d)\n", C_TAG, __FILE__, __LINE__, __FUNCTION__, ARG0, ARG1, ARG2)


using std::endl;

std::ofstream outFile;


class Detector {
    public:
        Detector();
        void process(int *pIn, int *pOut, int n);

    private:
        int m_var;
};


Detector::Detector()
: m_var(25) {
    outFile.open("addr_debug.txt");
    outFile << "m_var init address: " << &m_var << endl;
    PRINT_MSG_3SPI("&m_var(m_var)", &m_var, m_var);
}

void Detector::process(int *pIn, int *pOut, int n)
{
    outFile << "m_var process address: " << &m_var << endl;
    outFile.close();
    PRINT_MSG_3SPI("&m_var(m_var)", &m_var, m_var);
}


#if defined(__cplusplus)
extern "C" {
#endif

SIM_EXPORT Detector* DetectorNew() { return new Detector(); }

SIM_EXPORT void DetectorProcess(Detector *pDet, int *pIn, int *pOut, int n)
{
    pDet->process(pIn, pOut, n);
}

SIM_EXPORT void DetectorDelete(Detector *pDet) { delete pDet; }

#if defined(__cplusplus)
}
#endif

code00.py:

#!/usr/bin/env python

import ctypes as ct
import sys

import numpy as np


IntPtr = ct.POINTER(ct.c_int)

sim_dll = ct.CDLL("./sim.dll")

detector_new_func = sim_dll.DetectorNew
detector_new_func.argtypes = ()
detector_new_func.restype = ct.c_void_p

detector_process_func = sim_dll.DetectorProcess
detector_process_func.argtypes = (ct.c_void_p, IntPtr, IntPtr, ct.c_int)
detector_process_func.restype = None

detector_delete_func = sim_dll.DetectorDelete
detector_delete_func.argtypes = (ct.c_void_p,)
detector_delete_func.restype = None


class Detector():
    def __init__(self):
        self.obj = detector_new_func()

    def process(self, pin, pout, n):
        detector_process_func(self.obj, pin, pout, n)

    def __del__(self):
        detector_delete_func(self.obj)


def main(*argv):
    detector = Detector()

    n = 1024
    a = np.arange(n, dtype=np.uint32)
    b = np.zeros(n, dtype=np.int32)

    aptr = a.ctypes.data_as(IntPtr)
    bptr = b.ctypes.data_as(IntPtr)

    detector.process(aptr, bptr, n)


if __name__ == "__main__":
    print("Python {:s} {:03d}bit on {:s}\n".format(" ".join(elem.strip() for elem in sys.version.split("\n")),
                                                   64 if sys.maxsize > 0x100000000 else 32, sys.platform))
    rc = main(*sys.argv[1:])
    print("\nDone.")
    sys.exit(rc)

Notes:

• As I stated at the beginning, the problem was argtypes and restype not being specified (e.g. for DetectorNew: comment detector_new_func.restype = ct.c_void_p, and you'll run into the problem again)

• Code in the question is missing parts (#includes, imports, ...), also there are some syntax errors, so it doesn't compile, and therefore doesn't follow [SO]: How to create a Minimal, Complete, and Verifiable example (mcve) guidelines. Please when make sure to have MCVE when asking

• The object that you allocate (new Detector()), must also be deallocated (otherwise, it will generate a memory leak), so I added a function (DetectorDelete - to do that), which is called from (Python) Detector's destructor

• Other (non critical) changes (identifiers renaming, a bit of refactoring, printing to stdout, ...)

Output:

(py35x64_tes1) e:\Work\Dev\StackOverflow\q052268294>"c:\Install\x86\Microsoft\Visual Studio Community\2015\vc\vcvarsall.bat" x64

(py35x64_test) e:\Work\Dev\StackOverflow\q052268294>dir /b
code00.py
detector.cpp

(py35x64_test) e:\Work\Dev\StackOverflow\q052268294>cl /nologo /DDLL /EHsc detector.cpp  /link /DLL /OUT:sim.dll
detector.cpp
   Creating library sim.lib and object sim.exp

(py35x64_test) e:\Work\Dev\StackOverflow\q052268294>dir /b
code00.py
detector.cpp
detector.obj
sim.dll
sim.exp
sim.lib

(py35x64_test) e:\Work\Dev\StackOverflow\q052268294>"e:\Work\Dev\VEnvs\py35x64_test\Scripts\python.exe" ./code.py
Python 3.5.4 (v3.5.4:3f56838, Aug  8 2017, 02:17:05) [MSC v.1900 64 bit (AMD64)] 064bit on win32

From C - [detector.cpp] (28) - [Detector::Detector]:  &m_var: 0x0000020CE366E270
From C - [detector.cpp] (34) - [Detector::process]:  &m_var: 0x0000020CE366E270

Done.