Json.Net not deserializing references dynamically

Question

I have a large object graph with circular references, which I am serializing with Json.Net in order to preserve those references before sending to the client. On the client-side, I'm using a customized version of Ken Smith's JsonNetDecycle, which is, in turn, based on Douglas Crockford's cycle.js to restore the circular object references on deserialization, and remove them again before sending objects back to the server. On the server side, I'm using a custom JsonDotNetValueProvider similar to the one from this question in order to use Json.Net instead of the stock MVC5 JavaScriptSerializer. Everything seems to be working just fine from the server to the client and back again, with the Json surviving the round-trip just fine, but MVC won't deserialize the object graph correctly.

I've traced the problem down to this. When I use JsonConvert.Deserialize with a concrete type parameter, everything works, and I get a complete object graph with children and siblings properly referencing each other. This won't work for an MVC ValueProvider though, because you don't know the model type at that point in the lifecycle. The ValueProvider is just supposed to provide values in the form of a dictionary for the ModelBinder to use.

It appears to me that unless you can provide a concrete type for deserialization, the first reference to any given object in the graph will deserialize just fine, but any subsequent references to that same object will not. There's an object there, but it has none of its properties filled in.

To demonstrate, I've created the smallest demonstration I can of the problem. In this class (using Json.Net and NUnit), I create an object graph, and attempt to deserialize it in three different ways. See additional comments inline.

using System.Collections.Generic;
using System.Dynamic;
using System.IO;
using Newtonsoft.Json;
using Newtonsoft.Json.Converters;
using NUnit.Framework;

namespace JsonDotNetSerialization
{
[TestFixture]
public class When_serializing_and_deserializing_a_complex_graph
{
    public Dude TheDude;
    public Dude Gramps { get; set; }
    public string Json { get; set; }

    public class Dude
    {
        public List<Dude> Bros { get; set; }
        public string Name { get; set; }
        public Dude OldMan { get; set; }
        public List<Dude> Sons { get; set; }

        public Dude()
        {
            Bros = new List<Dude>();
            Sons = new List<Dude>();
        }
    }

    [SetUp]
    public void SetUp()
    {
        Gramps = new Dude
        {
            Name = "Gramps"
        };

        TheDude = new Dude
        {
            Name = "The Dude",
            OldMan = Gramps
        };

        var son1 = new Dude {Name = "Number one son", OldMan = TheDude};
        var son2 = new Dude {Name = "Lil' Bro", OldMan = TheDude, Bros = new List<Dude> {son1}};
        son1.Bros = new List<Dude> {son2};
        TheDude.Sons = new List<Dude> {son1, son2};
        Gramps.Sons = new List<Dude> {TheDude};

        var jsonSerializerSettings = new JsonSerializerSettings
        {
            PreserveReferencesHandling = PreserveReferencesHandling.Objects
        };

        Json = JsonConvert.SerializeObject(TheDude, jsonSerializerSettings);
    }

    [Test]
    public void Then_the_expected_json_is_created()
    {
        const string expected = @"{""$id"":""1"",""Bros"":[],""Name"":""The Dude"",""OldMan"":{""$id"":""2"",""Bros"":[],""Name"":""Gramps"",""OldMan"":null,""Sons"":[{""$ref"":""1""}]},""Sons"":[{""$id"":""3"",""Bros"":[{""$id"":""4"",""Bros"":[{""$ref"":""3""}],""Name"":""Lil' Bro"",""OldMan"":{""$ref"":""1""},""Sons"":[]}],""Name"":""Number one son"",""OldMan"":{""$ref"":""1""},""Sons"":[]},{""$ref"":""4""}]}";
        Assert.AreEqual(expected, Json);
    }

    [Test]
    public void Then_JsonConvert_can_recreate_the_original_graph()
    {
        // Providing a concrete type results in a complete graph
        var dude = JsonConvert.DeserializeObject<Dude>(Json);

        Assert.IsTrue(GraphEqualsOriginalGraph(dude));
    }

    [Test]
    public void Then_JsonConvert_can_recreate_the_original_graph_dynamically()
    {
        dynamic dude = JsonConvert.DeserializeObject(Json);

        // Calling ToObject with a concrete type results in a complete graph
        Assert.IsTrue(GraphEqualsOriginalGraph(dude.ToObject<Dude>()));
    }

    [Test]
    public void Then_JsonSerializer_can_recreate_the_original_graph()
    {
        var serializer = new JsonSerializer();
        serializer.Converters.Add(new ExpandoObjectConverter());
        var dude = serializer.Deserialize<ExpandoObject>(new JsonTextReader(new StringReader(Json)));

        // The graph is still dynamic, and as a result, the second occurrence of "The Dude" 
        // (as the son of "Gramps") will not be filled in completely.
        Assert.IsTrue(GraphEqualsOriginalGraph(dude));
    }

    private static bool GraphEqualsOriginalGraph(dynamic dude)
    {
        Assert.AreEqual("The Dude", dude.Name);
        Assert.AreEqual("Gramps", dude.OldMan.Name);
        Assert.AreEqual(2, dude.Sons.Count);
        Assert.AreEqual("Number one son", dude.Sons[0].Name);
        Assert.AreEqual("Lil' Bro", dude.Sons[0].Bros[0].Name);

        // The dynamic graph will not contain this object
        Assert.AreEqual("Lil' Bro", dude.Sons[1].Name);
        Assert.AreEqual("Number one son", dude.Sons[1].Bros[0].Name);
        Assert.AreEqual(1, dude.Sons[0].Bros.Count);
        Assert.AreSame(dude.Sons[0].Bros[0], dude.Sons[1]);
        Assert.AreEqual(1, dude.Sons[1].Bros.Count);
        Assert.AreSame(dude.Sons[1].Bros[0], dude.Sons[0]);

        // Even the dynamically graph forced through ToObject<Dude> will not contain this object.
        Assert.AreSame(dude, dude.OldMan.Sons[0]);

        return true;
    }
}
}

The JSON:

{
   "$id":"1",
   "Bros":[

   ],
   "Name":"The Dude",
   "OldMan":{
      "$id":"2",
      "Bros":[

      ],
      "Name":"Gramps",
      "OldMan":null,
      "Sons":[
         {
            "$ref":"1"
         }
      ]
   },
   "Sons":[
      {
         "$id":"3",
         "Bros":[
            {
               "$id":"4",
               "Bros":[
                  {
                     "$ref":"3"
                  }
               ],
               "Name":"Lil' Bro",
               "OldMan":{
                  "$ref":"1"
               },
               "Sons":[

               ]
            }
         ],
         "Name":"Number one son",
         "OldMan":{
            "$ref":"1"
         },
         "Sons":[

         ]
      },
      {
         "$ref":"4"
      }
   ]
}

I've seen plenty of examples of using Json.Net in a custom ValueProvider in order to support exactly this scenario, and none of the solutions have worked for me. I think the key thing that's missing is that none of the examples I've seen deal with the intersection of deserializing into a dynamic or expando object AND having internal references.

Answer 1

After rubber ducking this problem with a co-worker, the above behavior makes sense to me.

Without knowing the type of the object it's deserializing, Json.Net really has no way of knowing that the Sons or Bros properties aren't meant to be a string properties containing "{"$ref": "1"}"... how could it? Of course it deserializes it wrong. It has to know the target type in order to know when to further deserialize the properties of the object.

You end up with a dynamic object with string properties containing the Json representation of the object references. When the model binder tries to use this dynamic object to set the values on the concrete type, it finds no matches, and you end up with an empty instance of the target.

Jason Butera's answer to this question ends up being the most viable solution. Even though the default ValueProvider has already tried (and failed) to deserialize the object into a dictionary for the ModelBinder to use, the ModelBinder can choose to ignore all of that, and pull the raw input stream off of the controller context. Since the ModelBinder does know the type that the Json is supposed to be deserialized into, it can provide this to the JsonSerializer. It can also make use of the more convenient JsonConvert.DeserializeObject method.

The final code looks like this:

public class JsonNetModelBinder : IModelBinder
{
    public object BindModel(ControllerContext controllerContext, ModelBindingContext bindingContext)
    {
        var stream = controllerContext.RequestContext.HttpContext.Request.InputStream;
        stream.Seek(0, SeekOrigin.Begin);
        var streamReader = new StreamReader(stream, Encoding.UTF8);
        var json = streamReader.ReadToEnd();
        return JsonConvert.DeserializeObject(json, bindingContext.ModelType);
    }
}

Jason Butera's answer uses an attribute to mark each controller action with the proper ModelBinder. I took a more global approach. In Global.asax, I register the custom ModelBinder for all of my ViewModels using a little reflection:

var jsonModelBinder = new JsonNetModelBinder();
var viewModelTypes = typeof(ViewModelBase).Assembly.GetTypes()
    .Where(x => x.IsSubclassOf(typeof(ViewModelBase)));
viewModelTypes.ForEach(x => ModelBinders.Binders.Add(x, jsonModelBinder));

It all seems to be working so far, and uses a lot less code than the ValueProvider route.

Answer 2

Json itself doesn't have references which is why Json.Net doesn't try to preserve/restore references by default. It would be very awkward if Json.NET tried to restore references every time it found $id or $ref attributes. I also suspect this would force the parser to change its parsing strategy, start storing deserialized objects and keys etc.

You have to set the appropriate deserialization setting as shown in Preserving Object References, eg:

var settings=new JsonSerializerSettings {
                 PreserveReferencesHandling = PreserveReferencesHandling.Objects 
             };
var  deserializedPeople = JsonConvert.DeserializeObject<List<Person>>(json,settings);

If you still have problems you should try very minimal tests, ie try with small JSon snippets and move to more complex ones. For example, does the documentation example work? If no, you probably have an old Json.NET version. If yes, try with a more complex example until you find what is bugging Json.NET.

It's far easier to find the problem if you make small changes to a text snippet each time, than try to debug the entire serialization/deserialization chain