JsonConvert.PopulateObject is making my values default or null when inside a custom serializable class

Question

So I serialize and then use JsonConvert.PopulateObject to populate a Json back into a ScriptableObject, and that system works fine. It works fine with lists, arrays, anything that's on the first level.

This is my class (ManagedSubData inherits from ScriptableObject):

public class BuildableData : ManagedSubData
    {
        [SerializeField, JsonIgnore] private BuildableControllerV2 prefab;
        [SerializeField, JsonProperty] private float heightMove = 2f;
        [SerializeField, JsonProperty] private bool isStatic;
        [SerializeField, JsonProperty] private int initialQuantity = 1;
        [SerializeField, JsonProperty] private bool hasMinigame = false;
        [SerializeField, JsonProperty] private UpgradeLevel[] upgradeLevels;
    }

So basicly all of this works fine, I can use JsonIgnore, JsonProperty, If I need to use unity classes like Vector3, I have custom serializers. All works according to plan.. Except for the last property, UpgradeLevel, which is a custom serializable class :

[Serializable]
    public class UpgradeLevel
    {
        [SerializeField] private GameObject visual;
        [SerializeField, JsonIgnore] private int cost;
        [SerializeField] private Vector3 test;
    }

All 3 of these parameters, because they are either tagged with JsonIgnore or not tagged at all, once I populate, will default to their default. So, visual will become null, cost will become 0, and test will become 0,0,0.

With test and cost, it isn't an immediate problem, as if I tag them with JsonProperty it correctly gets that value. But visual is a GameObject reference, I can't and don't want to handle it, so it becomes null everytime and I have no alternative.

So the question is : Why would my custom UpgradeLevel class have this different behaviour for its fields? A setting somewhere perhaps? Contract resolver ?

BTW : I don't want to have to code a converter for every custom class like UpgradeLevel, we need to use this tool widely in the project and that would be inefficient

EDIT: These are my settings :

public static JsonSerializerSettings defaultSettings = new JsonSerializerSettings(){
            Converters = CreateConverters(),
            Formatting = Formatting.None,
            PreserveReferencesHandling = PreserveReferencesHandling.None,
            MissingMemberHandling = MissingMemberHandling.Ignore,
            ObjectCreationHandling = ObjectCreationHandling.Replace,
            TypeNameHandling = TypeNameHandling.All,
            ContractResolver = new NoPropertiesContractResolver()
        };

Answer 1

You are trying to populate an array member from JSON and recursively populate each item in the array from the corresponding JSON array, matching items by index. Unfortunately, as explained in this answer to JsonSerializer.CreateDefault().Populate(..) resets my values, this isn't implemented. Instead, when populating collections, Json.NET

• Replaces the contents of read-only collections and arrays.

• Concatenates the contents of resizable collections such as List<T>.

You also have an additional problem that you are setting ObjectCreationHandling = ObjectCreationHandling.Replace which always replaces inner objects with fresh instances, thereby disabling recursive population entirely. You need to remove this setting.

The converter in the linked answer, ArrayMergeConverter, is designed to only be applied via attributes or for specific item types T because there is no guaranteed way to determine whether any given POCO can be populated. E.g.:

• A string cannot be populated.

• An class with one or more immutable properties cannot be fully populated.

• A struct with entirely mutable properties, however, can be populated.

Since you want to create a general purpose ArrayMergeConverter for all item types, I would suggest the following:

1. Add some heuristics to determine whether a given item type can probably be populated.

2. Introduce an attribute that allows for that to be overridden, and apply it to types as required.

With that in mind, create the following attribute and converter:

[AttributeUsage(AttributeTargets.Class | AttributeTargets.Interface | AttributeTargets.Struct, AllowMultiple = false, Inherited = true)]
public sealed class JsonCanPopulateAttribute : System.Attribute
{
    public JsonCanPopulateAttribute(bool canPopulate) => this.CanPopulate = canPopulate;
    public bool CanPopulate { get; init; }
}

public class AutomaticArrayMergeConverter : ArrayMergeConverter
{
    static Lazy<IContractResolver> DefaultResolver { get; } = new (() => JsonSerializer.Create().ContractResolver );
    
    readonly IContractResolver resolver;
    
    public AutomaticArrayMergeConverter() : this(DefaultResolver.Value) { }
    public AutomaticArrayMergeConverter(IContractResolver resolver) => this.resolver = resolver ?? throw new ArgumentNullException(nameof(resolver));
    
    public override bool CanConvert(Type objectType) =>
        objectType.IsArray && objectType.GetArrayRank() == 1 && resolver.CanPopulateType(objectType.GetElementType()!);
}

// Adapted from this answer https://stackoverflow.com/a/40432055/3744182
// To https://stackoverflow.com/questions/40422136/jsonserializer-createdefault-populate-resets-my-values
public class ArrayMergeConverter : JsonConverter
{
    public override object? ReadJson(JsonReader reader, Type objectType, object? existingValue, JsonSerializer serializer)
    {
        if (!objectType.IsArray)
            throw new JsonSerializationException(string.Format("Non-array type {0} not supported.", objectType));
        var contract = (JsonArrayContract)serializer.ContractResolver.ResolveContract(objectType);
        if (contract.IsMultidimensionalArray)
            throw new JsonSerializationException("Multidimensional arrays not supported.");
        if (reader.MoveToContentAndAssert().TokenType == JsonToken.Null)
            return null;
        else if (reader.TokenType != JsonToken.StartArray)
            throw new JsonSerializationException(string.Format("Invalid start token: {0}", reader.TokenType));
        var existingArray = existingValue as System.Array;
        IList list = new List<object>();
        while (reader.ReadToContentAndAssert().TokenType != JsonToken.EndArray)
        {
            switch (reader.TokenType)
            {
                case JsonToken.Null:
                    list.Add(null);
                    break;
                default:
                    // Add item to list
                    var existingItem = existingArray != null && list.Count < existingArray.Length ? existingArray.GetValue(list.Count) : null;
                    if (existingItem == null)
                        existingItem = serializer.Deserialize(reader, contract.CollectionItemType);
                    else
                        serializer.Populate(reader, existingItem);
                    list.Add(existingItem);
                    break;
            }
        }

        var array = (existingArray != null && existingArray.Length == list.Count ? existingArray : Array.CreateInstance(contract.CollectionItemType!, list.Count));
        list.CopyTo(array, 0);
        return array;
    }

    public override bool CanWrite => false;
    public override void WriteJson(JsonWriter writer, object? value, JsonSerializer serializer) => throw new NotImplementedException();
    public override bool CanConvert(Type objectType) => throw new NotImplementedException("This converter is meant to be applied via attributes only.");
}

public static partial class JsonExtensions
{
    public static bool CanPopulateType(this IContractResolver resolver, Type type)
    {
        if (type.GetCustomAttribute<JsonCanPopulateAttribute>() is {} attr)
            return attr.CanPopulate;
        // No attribute, so apply some heuristics.
        var elementContract = resolver.ResolveContract(type);
        if (elementContract.Converter != null)
            return false;
        if (elementContract is JsonObjectContract c)
            // Apply some heuristics to see whether all properties are equally readable and writable.
            return c.Properties.All(p => p.Readable == p.Writable);
        return false;
    }
    
    public static JsonReader AssertTokenType(this JsonReader reader, JsonToken tokenType) => 
        reader.TokenType == tokenType ? reader : throw new JsonSerializationException(string.Format("Unexpected token {0}, expected {1}", reader.TokenType, tokenType));
    
    public static JsonReader ReadToContentAndAssert(this JsonReader reader) =>
        reader.ReadAndAssert().MoveToContentAndAssert();

    public static JsonReader MoveToContentAndAssert(this JsonReader reader)
    {
        if (reader == null)
            throw new ArgumentNullException();
        if (reader.TokenType == JsonToken.None)       // Skip past beginning of stream.
            reader.ReadAndAssert();
        while (reader.TokenType == JsonToken.Comment) // Skip past comments.
            reader.ReadAndAssert();
        return reader;
    }

    public static JsonReader ReadAndAssert(this JsonReader reader)
    {
        if (reader == null)
            throw new ArgumentNullException();
        if (!reader.Read())
            throw new JsonReaderException("Unexpected end of JSON stream.");
        return reader;
    }
}

Then add [JsonCanPopulate(bool)] to your types as required, e.g.:

[Serializable, JsonCanPopulate(true)]
public class UpgradeLevel
{
    public UpgradeLevel(GameObject visual, int cost, Vector3 test) =>
        (this.visual, this.cost, this.test) = (visual, cost, test);
    
    [SerializeField] private GameObject visual;
    [SerializeField, JsonProperty] private int cost;
    [SerializeField, JsonProperty] private Vector3 test;
}

And then populate e.g. as follows

var resolver = new NoPropertiesContractResolver(); // Cache statically to improve performance
var converter = new AutomaticArrayMergeConverter(resolver);

var defaultSettings =  new JsonSerializerSettings(){
    Converters = { converter },
    Formatting = Formatting.None,
    PreserveReferencesHandling = PreserveReferencesHandling.None,
    MissingMemberHandling = MissingMemberHandling.Ignore,
    //ObjectCreationHandling = ObjectCreationHandling.Replace -- Removed
    TypeNameHandling = TypeNameHandling.All, // Be sure you need this, see https://stackoverflow.com/q/39565954 for why.
    ContractResolver = resolver, 
};          

JsonConvert.PopulateObject(json, buildableData, defaultSettings);

Notes:

• To improve performance, Newtonsoft recommends you cache and reuse your custom contract resolver.

• ArrayMergeConverter is designed specifically for arrays, which cannot be resized. It would be straightforward to create a similar converter for all List<T> types by populating the existing items in place, adding more as required, and removing any excess items afterward.

  Or if you don't want to merge onto existing List<T> instances but would prefer to clear them before populating, you could use the logic from CollectionClearingContractResolver from this answer by Zoltán Tamási to Clear collections before adding items when populating existing objects.

• Using TypeNameHandling.All may expose you to security risks. For details see TypeNameHandling caution in Newtonsoft Json.

Demo fiddle here.

Answer 2

Are you using the JsonUtility class from Unity? from my experience is better to use that instead of JsonConvert, especially for the results I had in the builds.

BTW I remember I had a similar issue, and by setting the fields to public it worked out, as stated by ChrisBD in the comments!

EDIT: If you are using custom classes, remember to set them as public, and Serializable!