Is there a way to deserialize arbitrary JSON using Serde without creating fine-grained objects?

Question

I have a JSON object that contains a few metadata keys and a large data payload. My service cares about the metadata for the purposes of logging and routing, but does not care about the payload, other than to pass the payload off to another service. I will never need to look inside the payload for any reason.

Right now, the payload is represented in my struct as aserde_json::Value. Through profiling, I've seen the (de)serialization of the Value takes a non-trivial amount of time.

Is there a mechanism in Serde where I can bundle around the payload without having to pay the cost of deserializing it into component values only to be required to re-serialize them later?

extern crate serde;
#[macro_use]
extern crate serde_derive;
extern crate serde_json;

#[derive(Serialize, Deserialize)]
struct DataBlob<'a> {
    id: &'a str,
    priority: u8,
    // payload: OpaqueValue,
}

fn main() {
    let input = r#"{
        "id": "cat",
        "priority": 42,
        "payload": [1, 2, 3, 4]
    }"#;

    let parsed = serde_json::from_str::<DataBlob>(input).expect("Could not deserialize");
    let output = serde_json::to_string(&parsed).expect("Could not serialize");

    assert!(output.contains("payload"));
}

So in other words, deserialize `id` and `priority`, but not `payload`, and then later re-serialize an object with different `id` and/or `priority`, but the same (still serialized) `payload`? — MutantOctopus, Mar 03 '18 at 08:44
@BHustus yes, that seems correct. Another usage would be to move the payload to a new struct, changing the amount and types of metadata or even to output the payload by itself without any metadata. — Shepmaster, Mar 03 '18 at 12:55
After skimming through the docs a bit, it *seems* like the only way to do this via serde_json would be to create a custom implementation of [Read](https://docs.serde.rs/serde_json/de/trait.Read.html) that converts the entire `payload` value into a json substring, which could be plugged into a [Deserializer](https://docs.serde.rs/serde_json/de/struct.Deserializer.html) with `new`, and then do something similar with `Write` and `Format` for serialization. I hesitate to post it as a full answer because I'm not too familiar with serde, myself. — MutantOctopus, Mar 03 '18 at 21:26
write your own custom serializing and deserializing function for that struct? Use standard serde_json for the ones you want to inspect, and just a blob of bytes (base 64 or otherwise) for the DataBlob? — Andrew Mackenzie, Mar 13 '18 at 17:05

Shepmaster · Accepted Answer · 2018-09-23T22:33:05.487

This was added in serde_json 1.0.29 as the type RawValue. It must be enabled using the raw_value feature and then placed behind a reference:

extern crate serde; // 1.0.79
#[macro_use]
extern crate serde_derive; // 1.0.79
extern crate serde_json; // 1.0.30, features = ["raw_value"]

#[derive(Serialize, Deserialize)]
struct DataBlob<'a> {
    id: &'a str,
    priority: u8,
    payload: &'a serde_json::value::RawValue,
}

fn main() {
    let input = r#"{
        "id": "cat",
        "priority": 42,
        "payload": [1, 2, 3, 4]
    }"#;

    let parsed = serde_json::from_str::<DataBlob>(input).expect("Could not deserialize");
    let output = serde_json::to_string(&parsed).expect("Could not serialize");

    assert!(output.contains("payload"));
}

Is there a way to deserialize arbitrary JSON using Serde without creating fine-grained objects?

1 Answers1

Linked