How to deserialize a nested big array

Question

I have a type Foo which contains a big array. With the help of serde_big_array, serde_derive, and serde_with, I can derive serialization and deserializalization.

use serde_big_array;
use serde_big_array::BigArray;
use serde_derive::{Deserialize, Serialize};

pub const SIZE: usize = 30000;

#[derive(Clone, Debug, Serialize, Deserialize)]
struct Foo {
    #[serde(with = "BigArray")]
    pub vals: [bool; SIZE],
}

This works fine.

However, when use this type in another structure I run into trouble.

#[derive(Clone, Debug, Serialize, Deserialize)]
struct Bar {
    field0: Foo,
}

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn deserialization_attempt_test() {
        let foo = Foo {
            vals: [false; SIZE],
        };
        let instance: Bar = Bar { field0: foo };
        let json = serde_json::to_string(&instance).unwrap();
        println!("PMD0");
        let s_back = serde_json::from_str::<Bar>(&json).unwrap();
        println!("PMD1");
    }
}

Running this tests with cargo t deserialization_attempt_test -- --nocapture prints PMD0 but not PMD1. The test fails with the message

thread 'tests::deserialization_attempt' has overflowed its stack
fatal runtime error: stack overflow

How do I implement Serialize and Deserialize for Bar?

I found the `serde_stacker` crate and followed the description in their README. https://github.com/dtolnay/serde-stacker But I don't know how to get that code to work through the `serialize`/`deserialize` methods. — Thorkil Værge, Jun 14 '22 at 14:50
Changing the type of `vals` from `[bool; 30_000]` to `Vec` of same size makes the deserialization work. So that's an option if you can live with reduced type safety. — Thorkil Værge, Jun 15 '22 at 08:06

score 1 · Answer 1 · answered Jun 14 '22 at 15:25

The problem seems to be that the stack (when running through cargo test) is too small to handle your test. Note that if you move deserialization_attempt_test into a main function and run it with cargo run it will work. Similarly if you reduce the array size to e.g. 2000 it will also work.

A workaround, although not that satsifying, is to manually set RUST_MIN_STACK when running cargo test. For instance, this will work:

RUST_MIN_STACK=8388608 cargo test

More information about the error can be found here and here. Here is the documentation on RUST_MIN_STACK.

score 0 · Accepted Answer · answered Jun 14 '22 at 17:30

If you wrap your JSON deserializer in a serde_stacker instance, then it works without setting the environment variable that @munksgaard suggests.

    #[test]
    fn serialization_bug_fix_attempt_test() {
        let foo = Foo {
            vals: [false; SIZE],
        };

        let instance: Bar = Bar { field0: foo };
        let json = serde_json::to_string(&instance).unwrap();
        println!("PMD0");
        let mut deserializer: serde_json::Deserializer<serde_json::de::StrRead> =
            serde_json::Deserializer::from_str(&json);
        let deserializer = serde_stacker::Deserializer::new(&mut deserializer);
        let value: Value = Value::deserialize(deserializer).unwrap();
        let target: Bar = from_value(value).unwrap();
        println!("PMD1");
    }

How to deserialize a nested big array

2 Answers2

Linked