How to serialise and deserialise BTreeMaps with arbitrary key types?

Question

This example code:

use std::collections::BTreeMap;
use serde::{Deserialize, Serialize};

#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq, PartialOrd, Ord)]
struct Foo {
    bar: String,
    baz: Baz
}

#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq, PartialOrd, Ord)]
enum Baz {
    Quux(u32),
    Flob,
}

#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq, PartialOrd, Ord)]
struct Bish {
    bash: u16,
    bosh: i8
}

fn main() -> std::io::Result<()> {
    let mut btree: BTreeMap<Foo, Bish> = BTreeMap::new();
    let foo = Foo {
        bar: "thud".to_string(),
        baz: Baz::Flob
    };
    let bish = Bish {
        bash: 1,
        bosh: 2
    };


    println!("foo: {}", serde_json::to_string(&foo)?);
    println!("bish: {}", serde_json::to_string(&bish)?);
    
    btree.insert(foo, bish);
    println!("btree: {}", serde_json::to_string(&btree)?);

    Ok(())
}

gives the runtime output/error:

foo: {"bar":"thud","baz":"Flob"}
bish: {"bash":1,"bosh":2}
Error: Custom { kind: InvalidData, error: Error("key must be a string", line: 0, column: 0) }

I've googled this, and found that the problem is that the serialiser would be trying to write:

{{"bar":"thud","baz":"Flob"}:{"bash":1,"bosh":2}}}

which is not valid JSON, as keys must be strings.

The internet tells me to write custom serialisers.

This is not a practical option, as I have a large number of different non-string keys.

How can I make serde_json serialise to (and deserialise from):

{"{\"bar\":\"thud\",\"baz\":\"Flob\"}":{"bash":1,"bosh":2}}

for arbitrary non-string keys in BTreeMap and HashMap?

Answer 1

Although OP decided not to use JSON in the end, I have written a crate that does exactly what the original question asked for: https://crates.io/crates/serde_json_any_key. Using it is as simple as a single function call.

Because this is StackOverflow and just a link is not a sufficient answer, here is a complete implementation, combining code from v1.1 of the crate with OP's main function, replacing only the final call to serde_json::to_string:

extern crate serde;
extern crate serde_json;
use serde::{Serialize, Deserialize};
use std::collections::BTreeMap;

mod serde_json_any_key {
  use std::any::{Any, TypeId};
  use serde::ser::{Serialize, Serializer, SerializeMap, Error};
  use std::cell::RefCell;
  struct SerializeMapIterWrapper<'a, K, V>
  {
    pub iter: RefCell<&'a mut (dyn Iterator<Item=(&'a K, &'a V)> + 'a)>
  }

  impl<'a, K, V> Serialize for SerializeMapIterWrapper<'a, K, V> where
    K: Serialize + Any,
    V: Serialize
  {
    fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error> where
      S: Serializer
    {
      let mut ser_map = serializer.serialize_map(None)?;
      let mut iter = self.iter.borrow_mut();
      // handle strings specially so they don't get escaped and wrapped inside another string
      if TypeId::of::<K>() == TypeId::of::<String>() {
        while let Some((k, v)) = iter.next() {
          let s = (k as &dyn Any).downcast_ref::<String>().ok_or(S::Error::custom("Failed to serialize String as string"))?;
          ser_map.serialize_entry(s, &v)?;
        }
      } else {
        while let Some((k, v)) = iter.next() {
          ser_map.serialize_entry(match &serde_json::to_string(&k)
          {
            Ok(key_string) => key_string,
            Err(e) => { return Err(e).map_err(S::Error::custom); }
          }, &v)?;
        }
      }
      ser_map.end()
    }
  }

  pub fn map_iter_to_json<'a, K, V>(iter: &'a mut dyn Iterator<Item=(&'a K, &'a V)>) -> Result<String, serde_json::Error> where
  K: Serialize + Any,
  V: Serialize
  {
    serde_json::to_string(&SerializeMapIterWrapper {
      iter: RefCell::new(iter)
    })
  }
}


#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq, PartialOrd, Ord)]
struct Foo {
    bar: String,
    baz: Baz
}

#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq, PartialOrd, Ord)]
enum Baz {
    Quux(u32),
    Flob,
}

#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq, PartialOrd, Ord)]
struct Bish {
    bash: u16,
    bosh: i8
}

fn main() -> std::io::Result<()> {
    let mut btree: BTreeMap<Foo, Bish> = BTreeMap::new();
    let foo = Foo {
        bar: "thud".to_string(),
        baz: Baz::Flob
    };
    let bish = Bish {
        bash: 1,
        bosh: 2
    };


    println!("foo: {}", serde_json::to_string(&foo)?);
    println!("bish: {}", serde_json::to_string(&bish)?);

    btree.insert(foo, bish);
    println!("btree: {}", serde_json_any_key::map_iter_to_json(&mut btree.iter())?);
    Ok(())
}

Output:

foo: {"bar":"thud","baz":"Flob"}
bish: {"bash":1,"bosh":2}
btree: {"{\"bar\":\"thud\",\"baz\":\"Flob\"}":{"bash":1,"bosh":2}}

Answer 2

After discovering Rusty Object Notation, I realised that I was pushing a RON-shaped peg into a JSON-shaped hole.

The correct solution was to use JSON for the interface with the outside world, and RON for human-readable local data storage.