How do I write a Serde Visitor to convert an array of arrays of strings to a Vec>?

Question

I need to deserialize a JSON into a struct that has a Vec<Vec<f64>> field. The JSON has strings for numbers so I need a custom deserializer to convert the strings to f64 during the deserialization.

A sample JSON that I'd like to deserialize:

{
  "values": [["2", "1.4"], ["8.32", "1.5"]]
}

My struct is this:

#[derive(Deserialize)]
struct Payload {
    #[serde(default, deserialize_with = "from_array_of_arrays_of_strs")]
    values: Vec<Vec<f64>>,
}

I saw you could probably do this with visitors in the examples of Serde, so I've implemented this visitor:

fn from_array_of_arrays_of_strs<'de, T, D>(deserializer: D) -> Result<Vec<Vec<f64>>, D::Error>
where
    T: Deserialize<'de>,
    D: Deserializer<'de>,
{
    struct F64Visitor(PhantomData<fn() -> Vec<Vec<f64>>>);

    impl<'de> Visitor<'de> for F64Visitor {
        type Value = Vec<Vec<f64>>;

        fn expecting(&self, formatter: &mut fmt::Formatter) -> fmt::Result {
            formatter.write_str("a nonempty sequence of numbers")
        }

        #[inline]
        fn visit_str<E>(self, value: &str) -> Result<f64, E>
        where
            E: serde::de::Error,
        {
            self.visit_string(String::from(value))
        }

        #[inline]
        fn visit_string<E>(self, value: String) -> Result<f64, E> {
            Ok(value.parse::<f64>().unwrap())
        }

        #[inline]
        fn visit_seq<V, T>(self, mut visitor: V) -> Result<Vec<T>, V::Error>
        where
            V: SeqAccess<'de>,
        {
            let mut vec = Vec::new();

            while let Some(elem) = try!(visitor.next_element()) {
                vec.push(elem);
            }

            Ok(vec)
        }
    }

    let visitor = F64Visitor(PhantomData);
    deserializer.deserialize_seq(visitor)
}

playground

The compiler complains that visit_str and visit_string have an incompatible type for the trait:

error[E0053]: method `visit_str` has an incompatible type for trait
  --> src/main.rs:32:9
   |
32 | /         fn visit_str<E>(self, value: &str) -> Result<f64, E>
33 | |             where
34 | |             E: serde::de::Error,
35 | |         {
36 | |             self.visit_string(String::from(value))
37 | |         }
   | |_________^ expected struct `std::vec::Vec`, found f64
   |
   = note: expected type `fn(from_array_of_arrays_of_strs::F64Visitor, &str) -> std::result::Result<std::vec::Vec<std::vec::Vec<f64>>, E>`
              found type `fn(from_array_of_arrays_of_strs::F64Visitor, &str) -> std::result::Result<f64, E>`

error[E0053]: method `visit_string` has an incompatible type for trait
  --> src/main.rs:40:9
   |
40 | /         fn visit_string<E>(self, value: String) -> Result<f64, E> {
41 | |             Ok(value.parse::<f64>().unwrap())
42 | |         }
   | |_________^ expected struct `std::vec::Vec`, found f64
   |
   = note: expected type `fn(from_array_of_arrays_of_strs::F64Visitor, std::string::String) -> std::result::Result<std::vec::Vec<std::vec::Vec<f64>>, E>`
              found type `fn(from_array_of_arrays_of_strs::F64Visitor, std::string::String) -> std::result::Result<f64, E>`

error[E0049]: method `visit_seq` has 2 type parameters but its trait declaration has 1 type parameter
  --> src/main.rs:45:21
   |
45 |         fn visit_seq<V, T>(self, mut visitor: V) -> Result<Vec<T>, V::Error>
   |                     ^^^^^^ found 2 type parameters, expected 1

I think I don't have the correct understanding of how visitors work. Can I have only one visitor for deserializing the array of arrays of strings, or do I need one visitor for deserializing the arrays and one visitor for deserializing the strings to f64?

I've read:

score 9 · Accepted Answer · answered Jan 17 '18 at 03:29

As already described in How to transform fields before deserialization using serde?, the easiest solution is to introduce a newtype for your string-as-a-floating-point-value. You can then implement Deserialize for that, leveraging existing implementations of Deserialize and string parsing:

extern crate serde;
#[macro_use]
extern crate serde_derive;
extern crate serde_json;

use serde::de::{Deserialize, Deserializer, Error, Unexpected};

#[derive(Debug, Deserialize)]
struct Payload {
    #[serde(default)]
    values: Vec<Vec<Value>>,
}

#[derive(Debug)]
struct Value(f64);

impl<'de> Deserialize<'de> for Value {
    fn deserialize<D>(deserializer: D) -> Result<Value, D::Error>
        where D: Deserializer<'de>
    {
        let s: &str = Deserialize::deserialize(deserializer)?;
        s.parse()
            .map(Value)
            .map_err(|_| D::Error::invalid_value(Unexpected::Str(s), &"a floating point number as a string"))
    }
}

fn main() {
    let input = r#"
{
  "values": [["2", "1.4"], ["8.32", "1.5"]]
}
"#;

    let out: Payload = serde_json::from_str(input).unwrap();

    println!("{:?}", out);
}

I prefer this solution because in many cases I want that new type to play a role in my system.

If you really, truly need to deserialize once and to exactly a Vec<Vec<f64>>, you have to implement two visitors. One will deserialize the outer Vec, one will deserialize the inner Vec. We will reuse the previous Value newtype, but the inner visitor will strip it away. The outer visitor will do the same thing for a newtype around the inner visitor:

extern crate serde;
#[macro_use]
extern crate serde_derive;
extern crate serde_json;

use serde::de::{Deserialize, Deserializer, Error, SeqAccess, Unexpected, Visitor};
use std::fmt;

#[derive(Debug, Deserialize)]
struct Payload {
    #[serde(default, deserialize_with = "from_array_of_arrays_of_strs")]
    values: Vec<Vec<f64>>,
}

fn from_array_of_arrays_of_strs<'de, D>(deserializer: D) -> Result<Vec<Vec<f64>>, D::Error>
where
    D: Deserializer<'de>,
{
    struct OuterVisitor;

    impl<'de> Visitor<'de> for OuterVisitor {
        type Value = Vec<Vec<f64>>;

        fn expecting(&self, formatter: &mut fmt::Formatter) -> fmt::Result {
            formatter.write_str("a nonempty sequence of a sequence of numbers")
        }

        #[inline]
        fn visit_seq<V>(self, mut visitor: V) -> Result<Self::Value, V::Error>
        where
            V: SeqAccess<'de>,
        {
            let mut vec = Vec::new();

            while let Some(Inner(elem)) = try!(visitor.next_element()) {
                vec.push(elem);
            }

            Ok(vec)
        }
    }

    deserializer.deserialize_seq(OuterVisitor)
}

struct Inner(Vec<f64>);

impl<'de> Deserialize<'de> for Inner {
    fn deserialize<D>(deserializer: D) -> Result<Inner, D::Error>
    where
        D: Deserializer<'de>,
    {
        struct InnerVisitor;

        impl<'de> Visitor<'de> for InnerVisitor {
            type Value = Inner;

            fn expecting(&self, formatter: &mut fmt::Formatter) -> fmt::Result {
                formatter.write_str("a nonempty sequence of numbers")
            }

            #[inline]
            fn visit_seq<V>(self, mut visitor: V) -> Result<Inner, V::Error>
            where
                V: SeqAccess<'de>,
            {
                let mut vec = Vec::new();

                while let Some(Value(elem)) = try!(visitor.next_element()) {
                    vec.push(elem);
                }

                Ok(Inner(vec))
            }
        }

        deserializer.deserialize_seq(InnerVisitor)
    }
}

struct Value(f64);

impl<'de> Deserialize<'de> for Value {
    fn deserialize<D>(deserializer: D) -> Result<Value, D::Error>
    where
        D: Deserializer<'de>,
    {
        let s: &str = Deserialize::deserialize(deserializer)?;
        s.parse().map(Value).map_err(|_| {
            D::Error::invalid_value(Unexpected::Str(s), &"a floating point number as a string")
        })
    }
}

fn main() {
    let input = r#"
{
  "values": [["2", "1.4"], ["8.32", "1.5"]]
}
"#;

    let out: Payload = serde_json::from_str(input).unwrap();

    println!("{:?}", out);
}

score 3 · Answer 2 · answered Apr 25 '21 at 13:45

Parsing JSON files with strings instead of numbers is also possible without writing a visitor yourself.

use serde_with::{serde_as, DisplayFromStr};

#[serde_as]
#[derive(Debug, serde::Deserialize)]
struct Payload {
    #[serde_as(as = "Vec<Vec<DisplayFromStr>>")]
    #[serde(default)]
    values: Vec<Vec<f64>>,
}

let j = serde_json::json!({
  "values": [["2", "1.4"], ["8.32", "1.5"]]
});

let p: Payload = serde_json::from_value(j)?;
assert_eq!(p.values, vec![vec![2.0, 1.4], vec![8.32, 1.5]]);

The annotation means that we have a Vec inside a Vec and the innermost elements should be deserialized using FromStr and serialized using Display. The annotation is supports any type with Display and FromStr implementations, so it can also be used on u64 or on Url types.

this is much more concise than the other answer. Is this expected to be as performant as the previous answer? — financial_physician, Jan 22 '23 at 09:52
The performance should be pretty identical to the accepted answer. One advantage of this solution is that it avoids deserializing `&str`, which is problematic in a couple of cases. You also get the serialization for free. — jonasbb, Jan 22 '23 at 10:13

How do I write a Serde Visitor to convert an array of arrays of strings to a Vec>?

2 Answers2

Linked