Newtonsoft Json.NET JsonConverter attribute preserve references issue when deserializing

Question

In the models of a project I am using a JsonConverter attribute to help with the (de)serialization of those models.

The converter currently looks like this:

public class CustomJsonConverter : Newtonsoft.Json.JsonConverter
{
    bool _canWrite = true;
    public override bool CanWrite
    {
        get { return _canWrite; }
    }

    public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer)
    {
        serializer.PreserveReferencesHandling = PreserveReferencesHandling.Objects;
        serializer.DefaultValueHandling = DefaultValueHandling.Ignore;
        serializer.NullValueHandling = NullValueHandling.Ignore;

        _canWrite = false;
        var jObject = JObject.FromObject(value, serializer);
        _canWrite = true;

        jObject.WriteTo(writer);
    }

    public override object ReadJson(JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer)
    {
        serializer.PreserveReferencesHandling = PreserveReferencesHandling.Objects;
        if (reader.TokenType == JsonToken.StartObject)
        {
            existingValue = existingValue ?? serializer.ContractResolver.ResolveContract(objectType).DefaultCreator();
            serializer.Populate(reader, existingValue);
            return existingValue;
        }
        else if (reader.TokenType == JsonToken.Null)
        {
            return null;
        }
        else
        {
            throw new JsonSerializationException();
        }
    }

    public override bool CanConvert(Type objectType)
    {
        return typeof(IModelBase).IsAssignableFrom(objectType);
    }
}

The models have a base class that looks like this:

[JsonConverter(typeof(CustomJsonConverter))]
public abstract class ModelBase : IModelBase
{
    public string ID { get; set; }
    public DateTime CreatedAt { get; set; }
    public DateTime ModifiedAt { get; set; }
}

Derived classes of the ModelBase class have properties of which the type are also derived from ModelBase. For example:

public class CustomerModel : ModelBase
{
    public string Name { get; set; }
    public UserModel CreatedBy { get; set; }
    public UserModel ModifiedBy { get; set; }
}

public class UserModel : ModelBase
{
    public string Name { get; set; }
    public UserModel CreatedBy { get; set; }
    public UserModel ModifiedBy { get; set; }
}

I am using these models in an ASP.NET Web API 2 application (server side) and in C# applications (client side). For most of the API calls, an array of models is returned. When serializing the models, things work as expected. However, when deserializing, only the first occurrence of every reference is filled with information.

For example:

[
    {
        "$id": "1",
        "Name": "Customer1",
        "CreatedBy": {
            "$id": "2",
            "ID": "1",
            "Name": "User1"
        },
        "ModifiedBy": {
            "$id": "3",
            "ID": "3",
            "Name": "User3"
        },
        "ID": "1",
        "CreatedAt": "2019-02-06T00:00:04",
        "ModifiedAt": "2019-02-06T00:20:12"
    },
    {
        "$id": "4",
        "Name": "Customer2",
        "CreatedBy": {
            "$ref": "2"
        },
        "ModifiedBy": {
            "$ref": "2"
        },
        "ID": "2",
        "CreatedAt": "2019-02-06T00:10:00",
        "ModifiedAt": "2019-02-06T00:10:00"
    }
]

When trying to deserialize this JSON object returned by the web API, the CreatedBy and ModifiedBy properties will be correct for the first CustomerModel object. For the second CustomerModel object, however, those properties will be new UserModel instances without any properties set.

To deserialize the JSON string, I am using the following code:

using (var sr = new StreamReader(streamFromWebAPICall))
{                
    using (var jtr = new JsonTextReader(sr))
    {
        var js = new JsonSerializer();
        return js.Deserialize(jtr, objectType);
    }
}

What can I do to set the properties on all deserialized objects correctly?

Edit:

The problem seems to be in the serializer.Populate(reader, existingValue), where the references aren't remembered.

Answer 1

Your basic problem is that you are supplying a custom JsonConverter for a type for which you also want to enable PreserveReferencesHandling. But whenever a custom converter is applied, it must take care of everything manually, including parsing and generating of "$ref" and "$id" properties. Your converter does not do this, hence your deserialization code does not deserialize your object graph correctly.

The accepted answer to Custom object serialization vs PreserveReferencesHandling includes a template converter that shows how these properties can be dealt with. However, since your converter only seems to be toggling some serializer settings before (de)serialization, you could simply make a call to recursively (de)serialize the object, disabling the converter for the duration, like so:

public class CustomJsonConverter : Newtonsoft.Json.JsonConverter
{
    [ThreadStatic]
    static bool disabled;

    // Disables the converter in a thread-safe manner.
    bool Disabled { get { return disabled; } set { disabled = value; } }

    public override bool CanWrite { get { return !Disabled; } }

    public override bool CanRead { get { return !Disabled; } }

    public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer)
    {
        using (new PushValue<bool>(true, () => Disabled, val => Disabled = val))
        using (new PushValue<PreserveReferencesHandling>(PreserveReferencesHandling.Objects, () => serializer.PreserveReferencesHandling, val => serializer.PreserveReferencesHandling = val))
        using (new PushValue<DefaultValueHandling>(DefaultValueHandling.Ignore, () => serializer.DefaultValueHandling, val => serializer.DefaultValueHandling = val))
        using (new PushValue<NullValueHandling>(NullValueHandling.Ignore, () => serializer.NullValueHandling, val => serializer.NullValueHandling = val))
        {
            serializer.Serialize(writer, value);
        }
    }

    public override object ReadJson(JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer)
    {
        if (reader.TokenType == JsonToken.Null)
        {
            return null;
        }
        using (new PushValue<bool>(true, () => Disabled, val => Disabled = val))
        using (new PushValue<PreserveReferencesHandling>(PreserveReferencesHandling.Objects, () => serializer.PreserveReferencesHandling, val => serializer.PreserveReferencesHandling = val))
        using (new PushValue<DefaultValueHandling>(DefaultValueHandling.Ignore, () => serializer.DefaultValueHandling, val => serializer.DefaultValueHandling = val))
        using (new PushValue<NullValueHandling>(NullValueHandling.Ignore, () => serializer.NullValueHandling, val => serializer.NullValueHandling = val))
        {
            return serializer.Deserialize(reader, objectType);
        }
    }

    public override bool CanConvert(Type objectType)
    {
        throw new NotImplementedException();
    }
}

public struct PushValue<T> : IDisposable
{
    Action<T> setValue;
    T oldValue;

    public PushValue(T value, Func<T> getValue, Action<T> setValue)
    {
        if (getValue == null || setValue == null)
            throw new ArgumentNullException();
        this.setValue = setValue;
        this.oldValue = getValue();
        setValue(value);
    }

    // By using a disposable struct we avoid the overhead of allocating and freeing an instance of a finalizable class.
    public void Dispose()
    {
        if (setValue != null)
            setValue(oldValue);
    }
}

Note that the logic to disable the converter needs to be made thread-safe since Json.NET will share contracts and converters across threads.

Demo fiddle #1 here.

As an alternative, you could completely eliminate the converter and apply [JsonObject(IsReference = true)] directly to ModelBase:

[JsonObject(IsReference = true)]
public abstract class ModelBase : IModelBase
{
    public string ID { get; set; }
    public DateTime CreatedAt { get; set; }
    public DateTime ModifiedAt { get; set; }
}

Then serialize and deserialize with DefaultValueHandling.Ignore and NullValueHandling.Ignore specified in settings like so:

static object Deserialize(Stream streamFromWebAPICall, Type objectType)
{
    using (var sr = new StreamReader(streamFromWebAPICall))
    {
        using (var jtr = new JsonTextReader(sr))
        {
            var settings = new JsonSerializerSettings
            {
                DefaultValueHandling = DefaultValueHandling.Ignore,
                NullValueHandling = NullValueHandling.Ignore,
            };
            var js = JsonSerializer.CreateDefault(settings);
            return js.Deserialize(jtr, objectType);
        }
    }
}

Demo fiddle #2 here.

Note you can also set [JsonObject(ItemNullValueHandling = NullValueHandling.Ignore)] as of Json.NET 11.0.1, but there does not appear to be an ItemDefaultValueHandling setting on JsonObjectAttribute, so adding that to serializer settings (or using nullables for optional value-type values such as CreatedAt) is required.