How does ctypes map C struct members to Python class _fields_ when the mapping is not one-to-one?

Question

After reading this and this, I've tried writing my own "shared library" in C, and a Python wrapper to try and understand ctypes.

Notice that I've deliberately commented out member a in the Python wrapper to explore what happens when my mapping between the C struct's members and the Python class _fields_ are not exactly one-to-one:

testlib.c:

#include <stdio.h>
#include <stdlib.h>
#include <stdint.h>

void myprint();

void myprint() {
    printf("Hello world\n");
}

typedef struct object_t {
    uint8_t a;
    uint8_t b;
    uint8_t c;
    uint8_t d;
    uint8_t e;
    uint32_t f;
    uint8_t* g;
} object_t;

static object_t object  = {'a', 'b', 'c', 'd', 'e', 12345, NULL};

object_t* func1() {
    return &object; 
}

void func2(void(*callback)(object_t*), object_t* this_object) {
    callback(this_object);
}

I compile testlib.c into the shared library testlib.so in the same folder as follows:

gcc -shared -o testlib.so -fPIC testlib.c

wrapper.py:

from ctypes import *

testlib = CDLL('./testlib.so')
testlib.myprint()

class object_t(Structure):
    _fields_ = [
    # ('a', c_uint8),
    ('b', c_uint8),
    ('c', c_uint8),
    ('d', c_uint8),
    ('e', c_uint8),
    ('f', c_uint32),
    ('g', POINTER(c_uint8)),
    ]

callback_t = CFUNCTYPE(None, POINTER(object_t)) 

func1 = testlib.func1
func1.argtypes = None
func1.restype = POINTER(object_t) 

func2 = testlib.func2
func2.argtypes = [callback_t] 
func2.restype = None

ret = func1()

# a = ret.contents.a
b = ret.contents.b
c = ret.contents.c
d = ret.contents.d
e = ret.contents.e
f = ret.contents.f
g = ret.contents.g

print("{} {} {} {} {} {}".format(
    # chr(a),
    chr(b),
    chr(c),
    chr(d),
    chr(e),
    chr(f),
    g,
    ))

def mycallback(obj):
    # a = obj.contents.a
    b = obj.contents.b
    c = obj.contents.c
    d = obj.contents.d
    e = obj.contents.e
    f = obj.contents.f
    g = obj.contents.g
    print("{} {} {} {} {} {}".format(
    # chr(a),
    chr(b),
    chr(c),
    chr(d),
    chr(e),
    chr(f),
    g,
    ))

func2(callback_t(mycallback), ret)

And when I run python wrapper.py, I get the following output:

Hello world
a b c d e <__main__.LP_c_ubyte object at 0x7f503901cd08>
a b c d e <__main__.LP_c_ubyte object at 0x7f503902c048>

I see that this doesn't fail; instead, I'm still getting my first five letters, and then a pointer object.

Does the mapping simply correspond to the memory that each of the member types are supposed to take up, which then gets shoved into the type that I've specified in the Python field, regardless of whether or not it actually makes sense? (i.e. is my "12345" on the C side being put into POINTER(c_uint8) on the Python side)

The reason I ask is because I'm looking at the ctypes binding for PulseAudio in Kazam (https://bazaar.launchpad.net/~kazam-team/kazam/stable/view/head:/kazam/pulseaudio/ctypes_pulseaudio.py) and I see that many of the members in pa_source_info._fields_ are commented out (lines 119-135).

My guess is that if I want to uncomment additional items, ctypes would not handle the mapping correctly unless I uncommented down the list in a continuous chunk, and then also added classes/types for the currently unused type mappings like pa_usec_t or pa_source_port_info. Can someone please confirm? Thank you.

Antti Haapala -- Слава Україні · Answer 1 · 2018-10-21T18:05:40.027

My guess is that if I want to uncomment additional items, ctypes would not handle the mapping correctly unless I uncommented down the list in a continuous chunk, and then also added classes/types for the currently unused type mappings like pa_usec_t or pa_source_port_info.

Yes, that is exactly the case. ctypes tries to exactly follow the behaviour of the C compiler (but there has been some bugs with bitfields). A struct s in memory is just sizeof s contiguous bytes. To have ctypes produce correct layout, all the members must be at least of same size and have the same alignment requirements as the corresponding members in the C structure.

The C structure layout and internal padding is discussed in this question.

score 0 · Answer 2 · answered Oct 21 '18 at 16:52

It’s even worse than you think: part of a value may show up as the next member if the bytes don’t overlap exactly. And padding can make it non-trivial to predict what the misalignment will be: here, e probably has three bytes after it that have a good chance of being 0. Removing a also removed the padding, so that f in Python lines up with e in C.

How does ctypes map C struct members to Python class _fields_ when the mapping is not one-to-one?

2 Answers2