InvalidCastException JToken to JSON.net

Question

TL;DR: I was trying to create a class which would hold nested JSON data. I eventually solved my own problem, but @dbc was very helpful and they have a solution which may be slightly faster if you want to implement it their way. I have fully documented my solution, with example usage, and marked it as answered below.

I'm creating a project in which I intend to store lots of nested JSON data.

Instead of creating a hundred classes, each with their own variables/attributes, and then having to modify them every time I want to change something, I'd like to create a simple "dynamic object".

This object holds the root of all data, as well as all the children's data. In JSON, this is represented by:

{
    "name":"foo",
    "id":0,
    "attributes":
    {
        "slippery":true,
        "dangerous":true
    },
    "costs":
    {
        "move":1,
        "place":2,
        "destroy":3
    }
}

where the root structure holds the data "name" and "id", as well as children "attributes" and "costs" each containing their own data.

I'm using the json.net library for this, and my current class looks like this:

public class Data : JObject
{
    public void CreateChildUnderParent(string parent, string child)
    {
        Data obj = GetValueOfKey<Data>(parent);

        if(obj != null)
            obj.CreateChild(child);
    }

    public void CreateChild(string child)
    {
        AddKey(child, new Data());
    }

    public void AddKeyToParent(string parent, string key, JToken value)
    {
        Data parentObject = GetValueOfKey<Data>(parent);

        if(parentObject != null)
            parentObject.AddKey(key, value);
    }

    public void AddKey(string key, JToken value)
    {
        Add(key, value);
    }

    public void RemoveKeyFromParent(string parent, string key)
    {
        Data parentObject = GetValueOfKey<Data>(parent);

        if(parentObject != null)
            parentObject.RemoveKey(key);
    }

    public void RemoveKey(string key)
    {
        Remove(key);
    }

    public T GetValueFromParent<T>(string parent, string key)
    {
        Data parentObject = GetValueOfKey<Data>(parent);
        if(parentObject != null)
            return parentObject.GetValue(key).ToObject<T>();

        return default;
    }

    public T GetValueOfKey<T>(string key)
    {
        foreach (var kvp in this)
            if (kvp.Value is Data)
            {
                T value = ((Data)kvp.Value).GetValueOfKey<T>(key);
                if (value != null)
                    return value;
            }

        JToken token = GetValue(key);
        if(token != null)
            return token.ToObject<T>();  //throws exception

        return default;
    }
}

I can add children just fine, but my issue comes when I try to access them. An InvalidCastException is thrown within my

public T GetValueOfKey<T>(string key)

method whenever I call it using

Data 

as the generic type.

For example:

Data data = GetValueOfKey<Data>("attributes");

throws an exception. I'm not sure why this is happening, so any help would be greatly appreciated!

EDIT:

Here is the complete error log thrown:

InvalidCastException: Specified cast is not valid.
(wrapper castclass) System.Object.__castclass_with_cache(object,intptr,intptr)
Newtonsoft.Json.Linq.JToken.ToObject[T] () (at <97722d3abc9f4cf69f9e21e6770081b3>:0)
Data.GetValueOfKey[T] (System.String key) (at Assets/Scripts/Attributes/Object/Data.cs:74)
Data.AddKeyToParent (System.String parent, System.String key, Newtonsoft.Json.Linq.JToken value) (at Assets/Scripts/Attributes/Object/Data.cs:23)
DataController.Awake () (at Assets/Scripts/Controllers/DataController.cs:35)

and an example of instantiation which causes this exception:

public class DataController
{
    void Awake()
    {
        Data data = new Data();
        data.AddKey("name", "foo");
        data.CreateChild("attributes");
        data.AddKeyToParent("attributes", "slippery", true); //throws exception (line 35)
    }
}

UPDATE (10/20/18):

Ok so I went through my code this afternoon and rewrote it as a wrapper class, now the root JObject is stored within a variable in my Data, and accessor methods adjust its properties.

However, I ran into a problem. Here's the updated class (minified to the problem):

public class Data 
{
    public JObject data;

    public Data()
    {
        data = new JObject();
    }

    public void AddChild(string child)
    {
        data.Add(child, new JObject());
    }

    public void AddKeyWithValueToParent(string parent, string key, JToken value)
    {
        JObject parentObject = GetValueOfKey<JObject>(parent);

        if(parentObject != null)
            parentObject.Add(key, value);
    }

    public void AddKeyWithValue(string key, JToken value)
    {
        data.Add(key, value);
    }

    public T GetValueOfKey<T>(string key)
    {
        return GetValueOfKey<T>(key, data);
    }

    private T GetValueOfKey<T>(string key, JObject index)
    {
        foreach (var kvp in index)
            if (kvp.Value is JObject)
            {
                T value = GetValueOfKey<T>(key, kvp.Value.ToObject<JObject>());
                if (value != null)
                    return value;
            }

        JToken token = index.GetValue(key);
        if (token != null)
            return token.ToObject<T>();

        return default;
    }
}

And here is an example of how to construct a Data object, and use its methods:

public class DataController
{
    void Awake() {
        Data data = new Data();
        data.AddKeyWithValue("name", "foo");
        data.AddChild("attributes");
        data.AddKeyWithValueToParent("attributes", "slippery", true);
    }
}

So everything in terms of adding key-value pairs, and creating children works wonderfully! No InvalidCastException at all, yay! However, when I try to serialize the object through JsonConvert.SerializeObject(data), it doesn't fully serialize it.

I have the program output to the console to show the serialization, and it looks like this:

{"data":{"name":"foo","attributes":{}}}

I've already checked to make sure that when I call data.AddKeyWithValueToParent("attributes", "slippery", true), it does indeed find the JObject value with the key attributes and even appears to successfully add the new key-value pair "slippery":true under it. But for some reason, serializing the root object data does not seem to identify that anything lies within the attributes object. Thoughts?

What I think may be happening, is that the value returned from GetValueOfKey is not acting as a reference object, but rather an entirely new object, so changes to that are not reflected within the original object.

Answer 1

I figured it out! I was right, the value returned from my GetValueOfKey method was returning a completely new object, and not a reference to the instance it found. Looking through my code, that should have been immediately obvious, but I'm tired and I was hoping for everything to be easy haha.

Anyway, for anyone who ever has the same question, and is just looking for a simple way to store and read some nested key-value pairs using the Json.NET library, here is the finished class that will do that (also serializable, and deserializable using JsonConvert):

public class Data 
{
    [JsonProperty]
    private JObject data;

    public Data()
    {
        data = new JObject();
    }

    public void AddChildUnderParent(string parent, string child)
    {
        JObject parentObject = GetValueOfKey<JObject>(parent);

        if (parentObject != null)
        {
            parentObject.Add(child, new JObject());
            ReplaceObject(parent, parentObject);
        }
    }

    public void AddChild(string child)
    {
        data.Add(child, new JObject());
    }

    public void AddKeyWithValueToParent(string parent, string key, JToken value)
    {
        JObject parentObject = GetValueOfKey<JObject>(parent);

        if(parentObject != null)
        {
            parentObject.Add(key, value);
            ReplaceObject(parent, parentObject);
        }
    }

    public void AddKeyWithValue(string key, JToken value)
    {
        data.Add(key, value);
    }

    public void RemoveKeyFromParent(string parent, string key)
    {
        JObject parentObject = GetValueOfKey<JObject>(parent);

        if (parentObject != null)
        {
            parentObject.Remove(key);
            ReplaceObject(parent, parentObject);
        }
    }

    public void RemoveKey(string key)
    {
        data.Remove(key);
    }

    public T GetValueFromParent<T>(string parent, string key)
    {
        JObject parentObject = GetValueOfKey<JObject>(parent);

        if (parentObject != null)
            return parentObject.GetValue(key).ToObject<T>();

        return default;
    }

    public T GetValueOfKey<T>(string key)
    {
        return GetValueOfKey<T>(key, data);
    }

    private T GetValueOfKey<T>(string key, JObject index)
    {
        foreach (var kvp in index)
            if (kvp.Value is JObject)
            {
                T value = GetValueOfKey<T>(key, (JObject)kvp.Value);
                if (value != null)
                    return value;
            }

        JToken token = index.GetValue(key);
        if (token != null)
        {
            data = token.Root.ToObject<JObject>();
            return token.ToObject<T>();
        }

        return default;
    }

    public void ReplaceObject(string key, JObject replacement)
    {
        ReplaceObject(key, data, replacement);
    }

    private void ReplaceObject(string key, JObject index, JObject replacement)
    {
        foreach (var kvp in index)
            if (kvp.Value is JObject)
                ReplaceObject(key, (JObject)kvp.Value, replacement);

        JToken token = index.GetValue(key);
        if (token != null)
        {
            JToken root = token.Root;

            token.Replace(replacement);
            data = (JObject)root;
        }
    }
}

That should get anyone a good head start. I plan on updating my code with params modifiers in some places to allow for multiple calls, but for now I'm just happy that I got it working. You'll notice that I had to create a ReplaceObject method, because without it, the original private JObject data was never actually updated to account for the changes made to the variable returned from GetValueOfKey.

Anyway, a big thanks to @dbc for all their help during this whole thing, and I hope this post helps someone in the future!

-ShermanZero

EDIT:

So I spent a little more time developing the class, and I think I have it pinned down to a universal point where anyone could simply copy-paste and easily implement it into their own program. Although, I personally think that @dbc has a faster solution if you care about nanosecond-millisecond differences in speed. For my own personal use though, I don't think it will make much of a difference.

Here is my full implementation, complete with documentation and error logging:

public class Data 
{
    [JsonExtensionData]
    private JObject root;

    private Texture2D texture;

    private char delimiter = ',';

    /// <summary>
    /// Creates a new Data class with the default delimiter.
    /// </summary>
    public Data()
    {
        root = new JObject();
    }

    /// <summary>
    /// Creates a new Data class with a specified delimiter.
    /// </summary>
    /// <param name="delimiter"></param>
    public Data(char delimiter) : this()
    {
        this.delimiter = delimiter;
    }

    /// <summary>
    /// Adds a child node to the specified parent(s) structure, which is split by the delimiter, with the specified name.
    /// </summary>
    /// <param name="name"></param>
    /// <param name="parents"></param>
    public void AddChild(string name, string parents)
    {
        AddChild(name, parents.Split(delimiter));
    }

    /// <summary>
    /// Adds a child node to the specified parent(s) structure with the specified name.
    /// </summary>
    /// <param name="name"></param>
    /// <param name="parents"></param>
    public void AddChild(string name, params string[] parents)
    {
        string lastParent;
        JObject parentObject = ReturnParentObject(out lastParent, parents);

        if (parentObject != null)
        {
            parentObject.Add(name, new JObject());
            ReplaceObject(lastParent, parentObject, parents);
        } else
        {
            string message = "";
            foreach (string parent in parents)
                message += parent + " -> ";

            throw new ParentNotFoundException($"The parent '{ message.Substring(0, message.LastIndexOf("->")) }' was not found.");
        }
    }

    /// <summary>
    /// Adds a child node to the root structure with the specified name.
    /// </summary>
    /// <param name="name"></param>
    public void AddChild(string name)
    {
        root.Add(name, new JObject());
    }

    /// <summary>
    /// Adds the specified key-value pair to the specified parent(s) structure, which is split by the delimiter.
    /// </summary>
    /// <param name="key"></param>
    /// <param name="value"></param>
    /// <param name="parents"></param>
    public void AddKeyWithValue(string key, JToken value, string parents)
    {
        AddKeyWithValue(key, value, parents.Split(delimiter));
    }

    /// <summary>
    /// Adds the specified key-value pair to the specified parent(s) structure.
    /// </summary>
    /// <param name="key"></param>
    /// <param name="value"></param>
    /// <param name="parents"></param>
    public void AddKeyWithValue(string key, JToken value, params string[] parents)
    {
        string lastParent;
        JObject parentObject = ReturnParentObject(out lastParent, parents);

        if (parentObject != null)
        {
            parentObject.Add(key, value);
            ReplaceObject(lastParent, parentObject, parents);
        } else
        {
            string message = "";
            foreach (string parent in parents)
                message += parent + " -> ";

            throw new ParentNotFoundException($"The parent '{ message.Substring(0, message.LastIndexOf("->")) }' was not found.");
        }
    }

    /// <summary>
    /// Adds the specified key-value pair to the root structure.
    /// </summary>
    /// <param name="key"></param>
    /// <param name="value"></param>
    public void AddKeyWithValue(string key, JToken value)
    {
        root.Add(key, value);
    }

    /// <summary>
    /// Removes the specified key from the specified parent(s) structure, which is split by the delimiter.
    /// </summary>
    /// <param name="key"></param>
    /// <param name="parents"></param>
    public void RemoveKey(string key, string parents)
    {
        RemoveKey(key, parents.Split(delimiter));
    }

    /// <summary>
    /// Removes the specified key from the specified parent(s) structure.
    /// </summary>
    /// <param name="key"></param>
    /// <param name="parents"></param>
    public void RemoveKey(string key, params string[] parents)
    {
        string lastParent;
        JObject parentObject = ReturnParentObject(out lastParent, parents);

        if (parentObject != null)
        {
            parentObject.Remove(key);
            ReplaceObject(lastParent, parentObject, parents);
        } else
        {
            string message = "";
            foreach (string parent in parents)
                message += parent + " -> ";

            throw new ParentNotFoundException($"The parent '{ message.Substring(0, message.LastIndexOf("->")) }' was not found.");
        }
    }

    /// <summary>
    /// Removes the specified key from the root structure.
    /// </summary>
    /// <param name="key"></param>
    public void RemoveKey(string key)
    {
        root.Remove(key);
    }

    /// <summary>
    /// Returns if the specified key is contained within the parent(s) structure, which is split by the delimiter.
    /// </summary>
    /// <param name="key"></param>
    /// <param name="parents"></param>
    /// <returns></returns>
    public bool HasValue(string key, string parents)
    {
        return HasValue(key, parents.Split(delimiter));
    }

    /// <summary>
    /// Returns if the specified key is contained within the parent(s) structure.
    /// </summary>
    /// <param name="key"></param>
    /// <param name="parents"></param>
    /// <returns></returns>
    public bool HasValue(string key, params string[] parents)
    {
        //string lastParent = parents[parents.Length - 1];
        //Array.Resize(ref parents, parents.Length - 1);
        string lastParent;
        JObject parentObject = ReturnParentObject(out lastParent, parents);

        if (parentObject == null)
            return false;
        else if (parentObject == root && parents.Length > 0)
            return false;

        IDictionary<string, JToken> dictionary = parentObject;
        return dictionary.ContainsKey(key);
    }

    /// <summary>
    /// Returns the deepest parent object referenced by the parent(s).
    /// </summary>
    /// <param name="lastParent"></param>
    /// <param name="parents"></param>
    /// <returns></returns>
    private JObject ReturnParentObject(out string lastParent, string[] parents)
    {
        lastParent = null;
        if(parents.Length > 0)
        {
            lastParent = parents[parents.Length - 1];
            Array.Resize(ref parents, parents.Length - 1);

            return GetValueOfKey<JObject>(lastParent, parents);
        }

        return root;
    }

    /// <summary>
    /// Returns the value of the specified key from the specified parent(s) structure, which is split by the delimiter.
    /// </summary>
    /// <typeparam name="T"></typeparam>
    /// <param name="key"></param>
    /// <param name="parents"></param>
    /// <returns></returns>
    public T GetValueOfKey<T>(string key, string parents)
    {
        return GetValueOfKey<T>(key, parents.Split(delimiter));
    }

    /// <summary>
    /// Returns the value of the specified key from the specified parent(s) structure.
    /// </summary>
    /// <typeparam name="T"></typeparam>
    /// <param name="key"></param>
    /// <param name="parents"></param>
    /// <returns></returns>
    public T GetValueOfKey<T>(string key, params string[] parents)
    {
        JObject parentObject = null;
        for(int i = 0; i < parents.Length; i++)
            parentObject = GetValueOfKey<JObject>(parents[i].Trim(), parentObject == null ? root : parentObject);

        return GetValueOfKey<T>(key, parentObject == null ? root : parentObject);
    }

    /// <summary>
    /// Returns the value of the specified key from the root structure.
    /// </summary>
    /// <typeparam name="T"></typeparam>
    /// <param name="key"></param>
    /// <returns></returns>
    public T GetValueOfKey<T>(string key)
    {
        return GetValueOfKey<T>(key, root);
    }

    /// <summary>
    /// Returns the value of the specified key from a given index in the structure.
    /// </summary>
    /// <typeparam name="T"></typeparam>
    /// <param name="key"></param>
    /// <param name="index"></param>
    /// <returns></returns>
    private T GetValueOfKey<T>(string key, JObject index)
    {
        JToken token = index.GetValue(key);
        if (token != null)
            return token.ToObject<T>();

        foreach (var kvp in index)
            if (kvp.Value is JObject)
            {
                T value = GetValueOfKey<T>(key, (JObject)kvp.Value);
                if (value != null)
                    return value;
            }

        return default(T);
    }

    /// <summary>
    /// Replaces an object specified by the given key and ensures object is replaced within the correct parent(s), which is split by the delimiter.
    /// </summary>
    /// <param name="key"></param>
    /// <param name="replacement"></param>
    /// <param name="parents"></param>
    public void ReplaceObject(string key, JObject replacement, string parents)
    {
        ReplaceObject(key, root, replacement, parents.Split(delimiter));
    }

    /// <summary>
    /// Replaces an object specified by the given key and ensures object is replaced within the correct parent(s).
    /// </summary>
    /// <param name="key"></param>
    /// <param name="replacement"></param>
    /// <param name="parents"></param>
    public void ReplaceObject(string key, JObject replacement, params string[] parents)
    {
        ReplaceObject(key, root, replacement, parents);
    }

    /// <summary>
    /// Replaces an object specified by the given key.
    /// </summary>
    /// <param name="key"></param>
    /// <param name="replacement"></param>
    public void ReplaceObject(string key, JObject replacement)
    {
        ReplaceObject(key, root, replacement);
    }

    /// <summary>
    /// Replaces an object specified by the given key within the structure and updates changes to the root node.
    /// </summary>
    /// <param name="key"></param>
    /// <param name="index"></param>
    /// <param name="replacement"></param>
    private void ReplaceObject(string key, JObject index, JObject replacement)
    {
        foreach (var kvp in index)
            if (kvp.Value is JObject)
                ReplaceObject(key, (JObject)kvp.Value, replacement);

        JToken token = index.GetValue(key);
        if (token != null)
        {
            JToken root = token.Root;

            token.Replace(replacement);
            this.root = (JObject)root;
        }
    }

    /// <summary>
    /// Replaces an object specified by the given key within the structure, ensuring object is replaced within the correct parent, and updates changes to the root node.
    /// </summary>
    /// <param name="key"></param>
    /// <param name="index"></param>
    /// <param name="replacement"></param>
    /// <param name="parents"></param>
    private void ReplaceObject(string key, JObject index, JObject replacement, params string[] parents)
    {
        foreach (var kvp in index)
            if (kvp.Value is JObject)
            {
                bool valid = false;
                foreach (string str in parents)
                    if (str.Trim() == kvp.Key)
                        valid = true;

                if(valid)
                    ReplaceObject(key, (JObject)kvp.Value, replacement);
            }

        JToken token = index.GetValue(key);
        if (token != null)
        {
            JToken root = token.Root;

            token.Replace(replacement);
            this.root = (JObject)root;
        }
    }

    /// <summary>
    /// Returns the root structure as JSON.
    /// </summary>
    /// <returns></returns>
    public override string ToString()
    {
        return root.ToString();
    }

    /// <summary>
    /// A ParentNotFoundException details that the supplied parent was not found within the structure.
    /// </summary>
    private class ParentNotFoundException : Exception
    {
        public ParentNotFoundException() { }

        public ParentNotFoundException(string message) : base(message) { }

        public ParentNotFoundException(string message, Exception inner) : base(message, inner) { }
    }
}

Usage example:

Data data = new Data();

data.AddKeyWithValue("name", "foo");
data.AddChild("costs");
data.AddChild("attributes");

data.AddKeyWithValue("move", 1, "costs");
data.AddKeyWithValue("place", 2, "costs");
data.AddKeyWithValue("destroy", 3, "costs");

data.AddChild("movement", "costs");
data.AddKeyWithValue("slippery", false, "costs", "movement");

data.AddChild("movement", "attributes");
data.AddKeyWithValue("slippery", true, "attributes", "movement");

if(data.HasValue("move", "costs")) {
    Debug.Log(data.GetValueOfKey<int>("move", "costs")
    Debug.Log(data);
}

And its output:

1
{
 "name": "foo",
  "costs": {
    "move": 1,
    "place": 2,
    "destroy": 3,
    "movement": {
      "slippery": false
    }
  },
  "attributes": {
    "movement": {
      "slippery": true
    }
  }
}