TL;DR: I have a ctype Structure with another Structure inside, which I convert (seemingly correct) to a python dictionary, then I attempt to dump it into YAML. However the value of the internal Structure is shown wrong.
Python version: 3.10 Pyyaml version: 6.0
Background: I am trying to make our internal configuration handling more user-friendly. Our configuration files are the serialized copy of data from a C structure, and until now it was changed manually via hexeditor. My plan is to make this process more human readable, in a short term reading and writing YAML files.
I am using Ctypes Structure to serialize and deserialize the data, mainly because I have to support bitfields. The Ctype Structure could be used as a template, without actual values in it.
I have simplified the code by removing irrelevant functions and shortening the structures.
class substruct_t(My_Structure):
_pack_ = 1
_fields_ = [
("app", c_uint8, 4),
("showScale", c_uint8, 2),
("showIdleTemp", c_uint8, 2),
("type", c_uint8),
]
class ee_struct(My_Structure):
_pack_ = 1
_fields_ = [
("txStatusDelay", c_uint8, 5),
("overrideHours", c_uint8, 3),
("manualSet", c_int16),
("tempOffset", c_int8),
("substruct", substruct_t),
("LogArr", (c_uint8*6)*3),
("frostTemp", c_int8),
("fileVer", c_uint8*4),
]
class eeprom_t(Union):
_fields_ = [("as_struct", ee_struct), ("as_bytes", c_uint8*29)]
def __str__(self) -> str:
return str(self.as_struct)
def get_as_dict(self):
return self.as_struct.as_dict()
def get_as_bytes(self):
return np.ndarray((29, ), 'b', self.as_bytes, order='C')
I have created a My_Structure class, which inherits from Ctypes Structure, and allows different representation of the data, including dict. This seems to work well.
# Child class of Structure with string and dictionary representation functions, unwrapping arrays
class My_Structure(Structure):
def __recursive_carray_get(self, value):
# Necessary recursive function, if value is ctype array
if hasattr(value, '__len__'):
rtn = list()
for i in range(value.__len__()):
rtn.append(self.__recursive_carray_get(value.__getitem__(i)))
else:
rtn = value
return rtn
def __handle_array_type__(self, type):
# example unformatted type: <class '__main__.c_ubyte_Array_6_Array_3'>
return StringBetween("'", "'", str(type)).split(".")[1]
def __repr__(self) -> str:
return str(self.as_dict())
def __str__(self) -> str:
values = ",\n".join(f"{name}={value['value']}" for name, value in self.as_dict().items())
return f"<{self.__class__.__name__}: {values}>"
def as_dict(self) -> dict:
return {field[0]: {'value': self.__recursive_carray_get(getattr(self, field[0])), 'type': self.__handle_array_type__(field[1])}
for field in self._fields_}
However when I want to dump this dict into a YAML file, the value of substruct within ee_sruct is dumped badly, like a python object. I do not understand why, as a typecheck shows it is still a dict.
### Dict representation of substructure:
{'value': {'app': {'value': 2, 'type': 'c_ubyte'}, 'showScale': {'value': 0, 'type': 'c_ubyte'}, 'showIdleTemp': {'value': 1, 'type': 'c_ubyte'}, 'type': {'value': 2, 'type': 'c_ubyte'}}, 'type': 'substruct_t'}
### pyyaml dump of entire structure:
txStatusDelay: {value: 8, type: c_ubyte}
overrideHours: {value: 1, type: c_ubyte}
manualSet: {value: 100, type: c_short}
tempOffset: {value: 0, type: c_byte}
substruct:
value: !!python/object/apply:_ctypes._unpickle
- !!python/name:__main__.substruct_t ''
- !!python/tuple
- {}
- !!binary |
QgI=
type: substruct_t
LogArr:
value:
- [1, 0, 0, 0, 0, 0]
- [2, 0, 0, 0, 0, 0]
- [3, 0, 0, 0, 0, 0]
type: c_ubyte_Array_6_Array_3
frostTemp: {value: 16, type: c_byte}
fileVer:
value: [65, 66, 104, 0]
type: c_ubyte_Array_4
If I use a hardcoded dict with the exact same contents I get from get_as_dict, everything works.
Apparently, the dump functions don't get the same data as what gets printed from get_as_dict. Why is that, and how can I fix it?
What I tried:
My first idea was to implement a recursive function to return dict for internal structures (similarly what I did for arrays), but I was not sure where to start, as substruct is already reported as dict, and using the string (hardcoded) representation works.
How to export a Pydantic model instance as YAML with URL type as string seemed like a good approach, but combining Structure and YAMLObject resulted in a metaclass conflict, which I was unable to resolve.
I tried to dump into Json or using ruamel.yml, both throw an exception, complaining about substruct_t.
Combining Dumper class with string representer to get exact required YAML output could be the right approach, however it looks quite complicated, and I am hoping, there is a more simple solution that I just overlooked.
I just found a dirty fix, following the steps:
- convert the dict from
get_as_dict()to a string - replace all
'characters to" - use
json.loads()on the string to create a new dict, and use that instead It works, but it just underlines my question, why are the two dicts different to the dumpers?
