How to write-to/read-from Ctypes variable length array of ints

Question

I came across this link but am still struggling to construct an answer.

This is what one of the complex structs that I have looks like. This is actually a deep nested struct within other structs :)

/*
* A domain consists of a variable length array of 32-bit unsigned integers.
* The domain_val member of the structure below is the variable length array.
* The domain_count is the number of elements in the domain_val array.
*/
typedef struct domain {
    uint32_t domain_count;
    uint32_t *domain_val;
} domain_t;

The test code in C is doing something like this:

uint32_t domain_seg[4] = { 1, 9, 34, 99 };
domain_val = domain_seg;

The struct defined in python is

class struct_domain(ctypes.Structure):
  _pack_ = True # source:False
  _fields_ = [
             ('domain_count', ctypes.c_uint32),
             ('PADDING_0', ctypes.c_ubyte * 4),
             ('domain_val', POINTER_T(ctypes.c_uint32)),
             ]

How to populate the domain_val in that struct ? Can I use a python list ?

I am thinking something along dom_val = c.create_string_buffer(c.sizeof(c.c_uint32) * domain_count) but then how to iterate through the buffer to populate or read the values ?

Will dom_val[0], dom_val[1] be able to iterate through the buffer with the correct length ? Maybe I need some typecast while iterating to write/read the correct number of bytes

Mark Tolonen · Accepted Answer · 2021-11-17T19:00:20.840

Here's one way to go about it:

import ctypes as ct

class Domain(ct.Structure):
    _fields_ = (('domain_count', ct.c_uint32),
                ('domain_val', ct.POINTER(ct.c_uint32)))

    def __init__(self, data):
        size = len(data)
        # Create array of fixed size, initialized with the data
        self.domain_val = (ct.c_uint32 * size)(*data)
        self.domain_count = size

    # Note you can slice the pointer to the correct length to retrieve the data.
    def __repr__(self):
        return f'Domain({self.domain_val[:self.domain_count]})'

x = Domain([1, 9, 34, 99])
print(x)

# Just like in C, you can iterate beyond the end
# of the array and create undefined behavior,
# so make sure to index only within the bounds of the array.
for i in range(x.domain_count):
    print(x.domain_val[i])

Output:

Domain([1, 9, 34, 99])
1
9
34
99

To make it safer, you could add a property that casts the pointer to single element to a pointer to sized-array of elements so length checking happens:

import ctypes as ct

class Domain(ct.Structure):
    _fields_ = (('_domain_count', ct.c_uint32),
                ('_domain_val', ct.POINTER(ct.c_uint32)))

    def __init__(self,data):
        size = len(data)
        self._domain_val = (ct.c_uint32 * size)(*data)
        self._domain_count = size

    def __repr__(self):
        return f'Domain({self._domain_val[:self._domain_count]})'

    @property
    def domain(self):
        return ct.cast(self._domain_val, ct.POINTER(ct.c_uint32 * self._domain_count)).contents

x = Domain([1, 9, 34, 99])
print(x)

for i in x.domain: # now knows the size
    print(i)

x.domain[2] = 44   # Can mutate the array,
print(x)           # and it reflects in the data.
x.domain[4] = 5    # IndexError!

Output:

Domain([1, 9, 34, 99])
1
9
34
99
Domain([1, 9, 44, 99])
Traceback (most recent call last):
  File "C:\demo\test.py", line 27, in <module>
    x.domain[4] = 5
IndexError: invalid index

Thanks @Mark for the complete solution :) Much appreciated. I should have mentioned that the struct in python is already defined by someone, in my case, and it is not part of a class. But this statement here is where I was stuck originally `self.domain_val = (ct.c_uint32 * size)(*data)` How is this actually working ? And how does slicing retrieve the correct values — mittal, Nov 17 '21 at 19:09
Ah, maybe now I see !! So what I was thinking with `create_string_buffer` that way, it was just a raw buffer with no notion of bytes of each data element. The way you did it was making `domain_val` an array like https://stackoverflow.com/a/17938106/3821298 in C. I wasn't aware of that kind of C array initialization — mittal, Nov 17 '21 at 19:24
@mittal There wasn't a lot to go in in your question. If you want to update it with the Python code you are dealing with I can make a more targeted answer. I don't know what you mean by "the struct in python is already defined by someone ... and it is not part of a class". A class is how you wrap a C structure in `ctypes` so I'm interested what that means. — Mark Tolonen, Nov 17 '21 at 22:57
Sorry for that confusion, I missed some important words from my statement. I meant `the struct in python is already defined by someone ... and other than the fields of the struct they haven't defined any method(__init__) to populate the fields`. But I can negotiate with them and modify it as it is much more usable. As I said, the missing piece was that array initialization style which is cleaner than `create_string_buffer` — mittal, Nov 18 '21 at 09:47
I have updated the post with more details and pointed question. — mittal, Nov 18 '21 at 09:56

How to write-to/read-from Ctypes variable length array of ints

1 Answers1