1

I am trying to do the following:

  1. Deserialise a struct of fields into another struct of different format.
  2. I am trying to do the transformation via an Hashmap as this seems to be the best suited way.
  3. I am required to do this as a part of transformation of a legacy system, and this being one of the intermediary phases.
  4. I have raised another question which caters to a subset of the same use-case, but do not think it gives enough overview, hence raising this question with more details and code. (Rust converting a struct o key value pair where the field is not none)

Will be merging both questions once I figure out how.

Scenario:

  • I am getting input via an IPC through a huge proto.
  • I am using prost to deserialise this data into a struct.
  • My deserialised struct has n no of fields and can increase as well.
  • I need to transform this deserialised struct into a key,value struct. (shown ahead).
  • The incoming data, will mostly have a majority of null keys .i.e out of n fields, most likely only 1,2 have values at a given time.

Input struct after prost deserialisation:

Using proto3 so unable to define optional fields. Ideally I would prefer a prost struct of options on every field. .i.e Option instead of string.

struct Data{
  int32 field1,
  int64 field2,
  string field3,
  ...
}

This needs to be transformed to a genric struct as below for further use:

struct TransformedData
{
string Key
string Value
}

So,

Data{
 field1: 20
 field2: null
 field3: null
 field4: null
 ....
}

becomes

TransformedData{
key:"field1"
Value: "20"
}

Methods I have tried:

Method1

  1. Add serde to the prost struct definiton and deserialise it into a map.
  2. Loop over each item in a map to get values which are non-null.
  3. Use these to create a struct

https://play.rust-lang.org/?version=stable&mode=debug&edition=2015&gist=9645b1158de31fd54976926c9665d6b4

This has its challenges:

  1. Each primitve data type has its own default values and needs to be checked for.
  2. Nested structs will result in object data type which needs to be handled.
  3. Need to iterate over every field of a struct to determine non null values.

1&2 can be mitigated by setting an Optional field(yet to figure out how in proto3 and prost)

But I am not sure how to get over 3. Iterating over a large struct to find non null fields is not scalable and will be expensive.

Method 2:

I am using prost reflects dynamic reflection to deserialise and get only specified value.
Doing this by ensuring each proto message has two extra fields:

  1. proto -> signifying the proto struct being used when serializing.
  2. key -> signifying the filed which has value when serializing.
        let fd = package::FILE_DESCRIPTOR_SET;
        let pool = DescriptorPool::decode(fd).unwrap();
        let proto_message: package::GenericProto = prost::Message::decode(message.as_ref()).expect("error when de-serialising protobuf message to struct");
        let proto = proto_message.proto.as_str();
        let key = proto_message.key.as_str();

Then using key , I derive the key from what looks to be a map implemented by prost:


                let message_descriptor = pool.get_message_by_name(proto).unwrap();
                let dynamic_message = DynamicMessage::decode(message_descriptor, message.as_ref()).unwrap();
                let data = dynamic_message.get_field_by_name(key).unwrap().as_ref().clone();

Here :

1.This fails when someone sends a struct with multiple fields filled.
2.This does not work for nested objects, as nested objects in turn needs to be converted to a map.

I am looking for the least expensive way to do the above.
I understand the solution I am looking for is pretty out there and I am taking a one size fits all approach.

Any pointers appreciated.

  • Beside the fact that honestly I don't really get your point here :) because it's a lot of text, I read something about 100 fields and you are looking for "the least expensive way", but you already found a way, right? Why don't you simply use it? 100 fields is literally nothing. Is it too slow? I'd recommend to make it fast later and start with getting some more Rust experience first. – Aitch Aug 06 '22 at 11:37
  • @Aitch true, need some more hands on with rust. The problem with what I have found is not sure how to deal with nested values when using reflection of prost. And to the looping method I have, it's not 100 fields, its ~6k fields and has potential to increase. – Abhilash Gopalakrishna Aug 06 '22 at 11:57
  • 2
    Can you use `serde_prost` and [implement](https://serde.rs/impl-deserialize.html) `serde::Deserialize` for your custom struct that has a map in it? – Aitch Aug 06 '22 at 12:15

0 Answers0