How to pass (rapidly updated) variable from ctypes C code to Python? Pass by reference?

Question

I'm trying to develop C code that will be compiled to a DLL (or .SO for linux) with the purpose of processing some external input at high speed and returning the result to Python for follow-on processing.

My question is: what's the best way to regularly return values (at >1000s of time a second) from a C function for use in Python?

I created a test case as follows:

//dummy_function.c
#include <stdio.h>
#include <Windows.h>

__declspec(dllexport) int runme() {
    int i;
    for (i=1; i<= 500; i++) {
        printf("int = %d \n", i);
        Sleep(100);  // Add some arbitrary delay for now
    }
}

NB, I import Windows.h to use a dummy time delay (simulating my real-world problem). This code could run on unix using unistd.h instead (according to: What is the proper #include for the function 'sleep' in C?)

This code is then compiled into a .dll by

gcc -shared -o dummy_function.dll dummy_function.c

and imported in Python by:

import ctypes
libc = ctypes.CDLL('dummy_function.dll')
libc.runme()  # Start running

When executing this Python code, it does print out the increasing integers. However, catching the printed output from C using Python and then processing it doesn't seem like a good way to do this (/not scalable for high speed).

Instead, I wonder if there's a way to more easily pass variables to Python from the DLL. I guess I don't want to use the return function in C as this exits the function.

I've read about passing by reference in C, so wonder if this is compatible with ctypes. Could I define some memory location that the C code writes into, and the Python then polls this? Or better, some form of queue / buffer to avoid missing events?

Any advice would be greatly appreciated, thanks!

Answer 1

Although you're familiar with the concepts, here's [Python 3.Docs]: ctypes - A foreign function library for Python.

Python code polling a memory area while the C code is running, implies more than one thread.
Here's an example.

dll.c:

#include <stdio.h>
#include <Windows.h>

#if defined(_WIN32)
#  define DLL_EXPORT __declspec(dllexport)
#else
#  define DLL_EXPORT
#endif

#define PRINT_MSG_2XI(ARG0, ARG1) printf("From C - [%s] (%d) - [%s]:  ARG0: 0x%016p, ARG1: %d\n", __FILE__, __LINE__, __FUNCTION__, ARG0, ARG1)


typedef struct Srtruct_ {
    int value;
    int sentinel;
} Struct;


DLL_EXPORT int func0(Struct *ptr) {
    int counter = 0;
    if (!ptr) {
        return -1;
    }
    while (ptr->sentinel) {
        ptr->value++;
        counter++;
        PRINT_MSG_2XI(ptr, ptr->value);
        Sleep(200);
    }
    return counter;
}

code.py:

#!/usr/bin/env python3

import sys
import ctypes
import time
import threading


DLL_NAME = "./dll.dll"


class Struct(ctypes.Structure):
    _fields_ = [
        ("value", ctypes.c_int),
        ("sentinel", ctypes.c_int),
    ]


def thread_func(func, arg0):
    ret = func(arg0)
    print("\nFrom Python - Function returned {:d}".format(ret))


def main():
    dll = ctypes.CDLL(DLL_NAME)
    func0 = dll.func0
    func0.argtypes = [ctypes.POINTER(Struct)]
    func0.restype = ctypes.c_int

    data = Struct(30, 1)
    t = threading.Thread(target=thread_func, args=(func0, data,))
    t.start()
    time.sleep(1)
    print("From Python - Monitored value: {:d}".format(data.value))
    time.sleep(1)
    print("From Python - Monitored value: {:d}".format(data.value))
    data.sentinel = 0
    time.sleep(0.5)


if __name__ == "__main__":
    print("Python {:s} on {:s}\n".format(sys.version, sys.platform))
    main()
    print("\nDone.")

Output:

[cfati@CFATI-5510-0:e:\Work\Dev\StackOverflow\q056197585]> sopr.bat
*** Set shorter prompt to better fit when pasted in StackOverflow (or other) pages ***

[prompt]> "c:\Install\x86\Microsoft\Visual Studio Community\2017\VC\Auxiliary\Build\vcvarsall.bat" x64
**********************************************************************
** Visual Studio 2017 Developer Command Prompt v15.9.11
** Copyright (c) 2017 Microsoft Corporation
**********************************************************************
[vcvarsall.bat] Environment initialized for: 'x64'

[prompt]> dir /b
code.py
dll.c

[prompt]> cl /nologo /DDLL dll.c  /link /NOLOGO /DLL /OUT:dll.dll
dll.c
   Creating library dll.lib and object dll.exp

[prompt]> dir /b
code.py
dll.c
dll.dll
dll.exp
dll.lib
dll.obj

[prompt]> "e:\Work\Dev\VEnvs\py_064_03.07.03_test0\Scripts\python.exe" code.py
Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 22:22:05) [MSC v.1916 64 bit (AMD64)] on win32

From C - [dll.c] (27) - [func0]:  ARG0: 0x00000152EB84CE90, ARG1: 31
From C - [dll.c] (27) - [func0]:  ARG0: 0x00000152EB84CE90, ARG1: 32
From C - [dll.c] (27) - [func0]:  ARG0: 0x00000152EB84CE90, ARG1: 33
From C - [dll.c] (27) - [func0]:  ARG0: 0x00000152EB84CE90, ARG1: 34
From C - [dll.c] (27) - [func0]:  ARG0: 0x00000152EB84CE90, ARG1: 35
From Python - Monitored value: 35
From C - [dll.c] (27) - [func0]:  ARG0: 0x00000152EB84CE90, ARG1: 36
From C - [dll.c] (27) - [func0]:  ARG0: 0x00000152EB84CE90, ARG1: 37
From C - [dll.c] (27) - [func0]:  ARG0: 0x00000152EB84CE90, ARG1: 38
From C - [dll.c] (27) - [func0]:  ARG0: 0x00000152EB84CE90, ARG1: 39
From C - [dll.c] (27) - [func0]:  ARG0: 0x00000152EB84CE90, ARG1: 40
From Python - Monitored value: 40

From Python - Function returned 10

Done.