I am trying to do the following:
- Deserialise a struct of fields into another struct of different format.
- I am trying to do the transformation via an Hashmap as this seems to be the best suited way.
- I am required to do this as a part of transformation of a legacy system, and this being one of the intermediary phases.
- I have raised another question which caters to a subset of the same use-case, but do not think it gives enough overview, hence raising this question with more details and code. (Rust converting a struct o key value pair where the field is not none)
Will be merging both questions once I figure out how.
Scenario:
- I am getting input via an IPC through a huge proto.
- I am using prost to deserialise this data into a struct.
- My deserialised struct has n no of fields and can increase as well.
- I need to transform this deserialised struct into a key,value struct. (shown ahead).
- The incoming data, will mostly have a majority of null keys .i.e out of n fields, most likely only 1,2 have values at a given time.
Input struct after prost deserialisation:
Using proto3 so unable to define optional fields. Ideally I would prefer a prost struct of options on every field. .i.e Option instead of string.
struct Data{
int32 field1,
int64 field2,
string field3,
...
}
This needs to be transformed to a genric struct as below for further use:
struct TransformedData
{
string Key
string Value
}
So,
Data{
field1: 20
field2: null
field3: null
field4: null
....
}
becomes
TransformedData{
key:"field1"
Value: "20"
}
Methods I have tried:
Method1
- Add serde to the prost struct definiton and deserialise it into a map.
- Loop over each item in a map to get values which are non-null.
- Use these to create a struct
This has its challenges:
- Each primitve data type has its own default values and needs to be checked for.
- Nested structs will result in object data type which needs to be handled.
- Need to iterate over every field of a struct to determine non null values.
1&2 can be mitigated by setting an Optional field(yet to figure out how in proto3 and prost)
But I am not sure how to get over 3. Iterating over a large struct to find non null fields is not scalable and will be expensive.
Method 2:
I am using prost reflects dynamic reflection to deserialise and get only specified value.
Doing this by ensuring each proto message has two extra fields:
- proto -> signifying the proto struct being used when serializing.
- key -> signifying the filed which has value when serializing.
let fd = package::FILE_DESCRIPTOR_SET;
let pool = DescriptorPool::decode(fd).unwrap();
let proto_message: package::GenericProto = prost::Message::decode(message.as_ref()).expect("error when de-serialising protobuf message to struct");
let proto = proto_message.proto.as_str();
let key = proto_message.key.as_str();
Then using key , I derive the key from what looks to be a map implemented by prost:
let message_descriptor = pool.get_message_by_name(proto).unwrap();
let dynamic_message = DynamicMessage::decode(message_descriptor, message.as_ref()).unwrap();
let data = dynamic_message.get_field_by_name(key).unwrap().as_ref().clone();
Here :
1.This fails when someone sends a struct with multiple fields filled.
2.This does not work for nested objects, as nested objects in turn needs to be converted to a map.
I am looking for the least expensive way to do the above.
I understand the solution I am looking for is pretty out there and I am taking a one size fits all approach.
Any pointers appreciated.