How to Serialize and Deserialize a giant Trie?

Question

I am trying to serialize and deserialize a Trie that I have in memory. I thought it's going to be as simple as:

string jsonString = JsonSerializer.Serialize(giantTrieObj);
JsonSerializer.Deserialize<SuffixTrie>(jsonString);

Seems that is not the case. I just got an empty object {} upon serialization.

Also, the Trie I am dealing with is massive (5-6GB), so I am looking for an efficient way.

Trie:

public class SuffixTrie
{
    public TrieNode root = new TrieNode();
    public string endSymbol = "#";
    public int Count = 0;
    public SuffixTrie(DataTable ruleDt)
    {
        PopulateSuffixTrieFrom(ruleDt);
    }
   /// more code

Theer's no SuffixTrie in the BCL. What is that `giantTrieObj`? If it has no properties, the serializer won't produce anything. As for efficiency, serializing to a string is *by definition* not efficient. At the very least you should serialize to a file stream — Panagiotis Kanavos, Aug 23 '21 at 17:41
@PanagiotisKanavos it's a custom implementation of Trie data structure. I have added some code. `SuffixTrie` is an instance of `SuffixTrie` class — SamuraiJack, Aug 23 '21 at 17:43
Fields are implementation details, not properties. Even when they're public they're still implementation details, not part of the object's API. Serializers will persist the properties, not the fields, unless you tell them to. Which is a *bad* idea, especially with a verbose format like JSON. You'll end up with internal data that's of no interest to anyone. Eg, you'll get ` "endSymbol":"#"` in the final JSON. PS where's the `TrieNode`? What does *that* contain? And what schema do you want the JSON string to have? — Panagiotis Kanavos, Aug 23 '21 at 17:44
you need to use properties instead of fields..... otherwise the serializer will not serialize them by default — Jonathan Alfaro, Aug 23 '21 at 17:49
@PanagiotisKanavos Unbelievable! I can not believe I could be so silly. Thank you, so much! — SamuraiJack, Aug 23 '21 at 17:55
You are using [tag:System.Text.Json] not [tag:json.net]: [How to use class fields with System.Text.Json.JsonSerializer?](https://stackoverflow.com/q/58139759/3744182). — dbc, Aug 23 '21 at 17:55
Also, it is unlikely you will be able to fit a 6GB trie into a JSON string in memory, see [What is the maximum possible length of a .NET string?](https://stackoverflow.com/q/140468/3744182). Instead you may need to serialize and deserialize to and from a `Stream` as shown in [How to deserialize stream to object using System.Text.Json APIs](https://stackoverflow.com/q/58512393/3744182). — dbc, Aug 23 '21 at 18:12
Instead of JSON you should probably use a more efficient format like BSON or Protocol Buffers. You can also wrap the FileStream in a GZipStream or BrotliStream to compress the data as it's written to the disk — Panagiotis Kanavos, Aug 23 '21 at 18:28
@dbc I serialized it and wrote it to file and it was a little over 250MB. But in memory, it does consume 6 GB. Perhaps because of creating Dictionary objects? — SamuraiJack, Aug 23 '21 at 18:50
@SamuraiJack - don't know, maybe. Could be due to the underlying arrays inside `Dictionary` being exponentially resized, could be due to utf8 vs utf16 encoding, could be something else, not enough information to say without a [mcve]. But can this question be closed as a duplicate of [How to use class fields with System.Text.Json.JsonSerializer?](https://stackoverflow.com/q/58139759/3744182)? — dbc, Aug 23 '21 at 19:38
@PanagiotisKanavos you should post your comment as an answer. I will mark it. — SamuraiJack, Aug 23 '21 at 19:41

How to Serialize and Deserialize a giant Trie?

0 Answers0