How can I use Serde's custom (de)serialization to update a subset of arbitrary input?

Question

I need to update specific fields of an arbitrary input file without touching any keys or values that my program does not know about.

Here is an example input file:

{ 
  "alpha": {
    "a": 1,
    "z": 2
  },
  "beta": "b"
}

I'd like to update alpha.a by 100:

{ 
  "alpha": {
    "a": 101,
    "z": 2
  },
  "beta": "b"
}

It is possible to do this with types like serde_json::Value and toml::value::Value, but this code is very cumbersome:

extern crate serde; // 1.0.66
extern crate serde_json; // 1.0.21

use serde_json::Value;

fn main() {
    let input = r#"{ 
      "alpha": {
        "a": 1,
        "z": 2
      },
      "beta": "b"
    }"#;

    let mut to_change: Value = serde_json::from_str(input).unwrap();
    {
        let obj = to_change.as_object_mut().unwrap();
        let alpha = obj.get_mut("alpha").unwrap();

        let obj = alpha.as_object_mut().unwrap();
        let num = {
            let a = obj.get("a").unwrap();

            let mut num = a.as_i64().unwrap();
            num += 100;
            num
        };
        obj.insert("a".into(), Value::Number(num.into()));
    }
    println!("{}", serde_json::to_string_pretty(&to_change).unwrap());
}

I'd much rather use the clean derived syntax:

extern crate serde; // 1.0.66
#[macro_use]
extern crate serde_derive; // 1.0.66
extern crate serde_json; // 1.0.21

#[derive(Debug, Deserialize, Serialize)]
struct WhatICareAbout {
    alpha: Alpha,
}

#[derive(Debug, Deserialize, Serialize)]
struct Alpha {
    a: i32,
}

fn main() {
    let input = r#"{ 
          "alpha": {
            "a": 1,
            "z": 2
          },
          "beta": "b"
        }"#;

    let mut subobject: WhatICareAbout = serde_json::from_str(input).unwrap();
    subobject.alpha.a += 1;
    println!("{}", serde_json::to_string_pretty(&subobject).unwrap());
}

This runs, but it strips out any unknown keys:

{
  "alpha": {
    "a": 2
  }
}

Is there a way I can use the pretty Deserialize and Serialize implementations while still preserving keys and values I don't know about?

An ideal answer would:

Work for most Serde formats — I'm showing JSON here but my real code is TOML.
Allow adding, updating, and removing fields.

Shepmaster · Accepted Answer · 2018-07-24T12:55:28.510

4

As of Serde 1.0.34, you can use #[serde(flatten)] to capture all the unrecognized keys and values you don't care about to create a "catch-all" field:

extern crate serde; // 1.0.66
#[macro_use]
extern crate serde_derive; // 1.0.66
extern crate serde_json; // 1.0.21

type Other = serde_json::Map<String, serde_json::Value>;

#[derive(Debug, Deserialize, Serialize)]
struct WhatICareAbout {
    alpha: Alpha,
    #[serde(flatten)]
    other: Other,
}

#[derive(Debug, Deserialize, Serialize)]
struct Alpha {
    a: i32,
    #[serde(flatten)]
    other: Other,
}

For TOML, you can use a parallel definition of Other:

type Other = std::collections::BTreeMap<String, Value>;

Am I correct in assuming that the order of keys/formatting will be completely discarded by this approach? (At the very least, I expect alpha to be serialized first, regardless of the position it originally occupied)

Yes, the order of the output keys will be based on a combination of the field order in the struct definition and the type of map you choose. Above, a BTreeMap is used, so the "other" keys will be alphabetized, but all come after the specific fields.

You could opt into an IndexMap which would preserve the order of the "other" keys, although they'd still come relative to the specific fields you've added:

extern crate indexmap; // 0.4.1 + features = ["serde-1"]

type Other = indexmap::IndexMap<String, serde_json::Value>;

See also:

edited Jul 24 '18 at 12:55

answered Jun 23 '18 at 15:29

Shepmaster

388,571
95
1,107
1,366

1

Am I correct in assuming that the order of keys/formatting will be completely discarded by this approach? (At the very least, I expect `alpha` to be serialized first, regardless of the position it originally occupied) If so, a note might be welcome, as using such a "script" on a committed file could result in large changes. – Matthieu M. Jun 23 '18 at 17:34
@MatthieuM. an interesting point! It would normally be relevant to my case of editing a Cargo.toml, which is usually a file committed to source control that is human-edited. In my case, it doesn't matter since it's for the playground. – Shepmaster Jun 23 '18 at 19:00
When I try this, I get ``error: unknown serde variant attribute `flatten``, even though I have Serde v1.0.70. It would be nice if you could add the necessary ``use`` statements to your answer. – Philipp Ludwig Jul 24 '18 at 08:26
@PhilippLudwig The crate statements and versions are in the question, but I can copy them if that's not obvious. – Shepmaster Jul 24 '18 at 12:54
@Shepmaster My bad, I thought that this feature could be used during serialization as well (e.g. to flatten JSON structures). In the end I've just simplified my data structure. – Philipp Ludwig Jul 25 '18 at 14:05
@PhilippLudwig yes, [that's exactly what I'm doing](https://play.rust-lang.org/?gist=2591d19e15336960c4ffedada9558728&version=stable&mode=debug&edition=2015). – Shepmaster Jul 25 '18 at 14:24

How can I use Serde's custom (de)serialization to update a subset of arbitrary input?

1 Answers1

Linked