.Net System.Text.Json: how to deserialize to a class constructor where parameters nullability doesn't match class properties nullability

Question

I have a bunch of Json files that I want to deserialize to immutable classes.

E.g. a Json file like this :

{
  "Prop1": 5,
  "Prop3": {
    "NestedProp1": "something",
    "NestedProp3": 42
  }
}

Should be deserialized to classes like that:

public class Outer 
{
  public int Prop1 { get; }
  public int Prop2 { get; }
  public Inner Prop3 { get; }
  public string Prop4 { get; }
  public Outer(int prop1, int? prop2, Inner prop3, string? prop4)
  {
    Prop1 = prop1;
    Prop2 = prop2 ?? GetSensibleRuntimeProp2Default();
    Prop3 = prop3;  
    Prop4 = prop4 ?? GetSensibleRuntimeProp4Default();
  }
...
}

public class Inner
{
  public string NestedProp1 { get; }
  public int NestedProp2 { get; }
  public int NestedProp3 { get; }
  public Inner(string nestedProp1, int? nestedProp2, int nestedProp3)
  {
    NestedProp1 = nestedProp1;
    NestedProp2 = nestedProp2 ?? GetSensibleRuntimeNestedProp2Default();
    NestedProp3 = nestedProp3;
  }
...
}

As one can see, some constructor parameters are nullable (ref or value types), so that some default value can be injected in case the value is not specified in the Json file. However the matching properties in the classes are not nullable.

The problem is System.Text.Json deserialization requires that constructor parameters and properties have the exact same type so if I try this I get an exception stating this requirement which is indeed documented.

How would I be able to work around this limitation? Could I somehow inject code in the deserilization process to insert my own policy for deserializing objects (while letting the default process handle values and arrays)?

I am dealing with existing classes and Json and I am not allowed to make changes like adding a nullable property to the classes in order to match the constructor parameter. The thing actually worked using Newtonsoft.Json and I am asked to convert it to using System.Text.Json.

I wrote come code that uses reflection to deserialize the first level object in the Json file using the target class constructor that best matches the Json object properties. It looks like that:

public static object CreateInstance(Type targetType, JsonObject prototype, JsonSerializerOptions serializerOptions)
{
    var constructors = GetEligibleConstructors(targetType);
    var prototypePropertySet = prototype.Select(kvp => kvp.Key)
          .ToImmutableHashSet(StringComparer.CurrentCultureIgnoreCase);
    var bestMatch = FindBestConstructorMatch(prototypePropertySet, constructors);
    if (bestMatch is null)
        throw new NoSuitableConstructorFoundException($"COuld not find a suitable constructor to instanciate {targetType.FullName} from \"{prototype.ToJsonString()}\"");
    var valuedParams = GetParameterValues(bestMatch, prototype, serializerOptions);
    return bestMatch.Constructor.Invoke(bestMatch.Parameters.Select(p => valuedParams[p.Name]).ToArray());
}

(I'm deserializing from a JsonObject and not from text but I don't think it's relevant to the issue)

So basically:

• get all public constructors of the target type,
• find the one that has the most parameters in common with the Json object properties
• get the parameters values from the Json using standard System.Text.Json deserialization
• Invoke the constructor

Obviously this method limits me to the first level object since all the child properties will be handled by standard deserialization.

I would like to be able to do that recursively by using a similar code in a JsonConverter that would be called on deserializing Json objects while the deserialization of Json arrays or primitive values would be left to the standard converters.

Answer 1

Following @dbc advice, I used the JsonConverterFactory pattern to address my issue.

Here are the two classes I wrote. The converter itself:

public class CustomObjectConverter<T> : JsonConverter<T> where T : class
{
    private record struct UnmatchedParameterInfo(string Name, Type Type, bool AcceptsNullValue);
    private record struct MatchedParameterInfo(string Name, Type Type, object Value);

    // ConstructorMatcher is a helper class that will keep track of 
    // Json properties that can be matched with the parameters of 
    // a specific constructor as they are read from the reader
    private class ConstructorMatcher
    {
        public ConstructorInfo ConstructorInfo { get; }
        private Dictionary<string, UnmatchedParameterInfo> UnmatchedParameters { get; }
        private Dictionary<string, MatchedParameterInfo> MatchedParameters { get; } = new();

        private ImmutableList<string> ParameterList { get; }

        public ConstructorMatcher(ConstructorInfo constructorInfo)
        {
            ConstructorInfo = constructorInfo ?? throw new ArgumentNullException(nameof(constructorInfo));
            var parameters = constructorInfo.GetParameters();
            if (parameters.Any(p => p.Name is null))
                throw new Exception("<useful exception message>");
            ParameterList = parameters.Select(p => p.Name!).ToImmutableList();
            UnmatchedParameters = 
                 parameters.ToDictionary(
                              p => p.Name!, 
                              p => new UnmatchedParameterInfo(
                                           p.Name!, 
                                           p.ParameterType,
                                           AcceptsNullValue(p.ParameterType)), 
                              StringComparer.CurrentCultureIgnoreCase);
        }

        // Checks is null can be assigned to a certain type
        private bool AcceptsNullValue(Type type)
        {
            // This is improvable as it doesn't use reference types 
            // nullability information
            return type.IsClass || Nullable.GetUnderlyingType(type) is not null;
        }

        public int MatchCount => MatchedParameters.Count;

        public int UnmatchedCount => UnmatchedParameters.Count;

        // Checks if the constructor still has a yet unmatched parameter
        // named "name"
        public bool HasParameter(string name) => UnmatchedParameters.ContainsKey(name);

        // Returns the type that the constructor expects for parameter "name".
        // The converter needs this because it has to know beforehand the type
        // of the property it is about to deserialize.
        public Type GetTypeForParameter(string name)
        {
            return UnmatchedParameters.TryGetValue(name, out var info) 
                       ? info.Type 
                       : throw new Exception("<useful exception message>");
        }

        // Binds a value to a constructor parameter
        public void AddParameterValue(string name, Type type, object? value)
        {
            if (!UnmatchedParameters.Remove(name, out var unmatchedParameterInfo))
                throw new Exception("<useful exception message>");
            if (unmatchedParameterInfo.Type != type)
                throw new Exception("<useful exception message>");
            MatchedParameters.Add(unmatchedParameterInfo.Name, 
                                  new MatchedParameterInfo(
                                         unmatchedParameterInfo.Name,
                                         unmatchedParameterInfo.Type, 
                                         value));
        }

        // Checks if the constructor has some unbound parameters
        // that won't accept null values
        public bool HasNonNullableUnmatchedParameters => UnmatchedParameters.Values.Any(upi => !upi.AcceptsNullValue);

        // Gets parameter values in the righ order for constructor invocation
        public object?[] GetInvocationParameters()
        {
            return ParameterList.Select(parameterName => MatchedParameters.TryGetValue(parameterName, out var mpi)
                                                             ? mpi.Value
                                                             : null)
                                .ToArray();
        }
    }

    // For each Json property:
    // - read the Json property name, 
    // - eliminate constructors that don't have a parameter with 
    //   a compatible name
    // - get the expected type for this property (end throw if all 
    //   constructors don't expect the same type for the same property)
    // - deserialize the property value to the expected type
    // - bind the deserialized value to the matching parameter of each 
    //   candidate constructor
    // When all properties are read, invoke the constructor that has the 
    // most parameters bound to values, with the less unbound parameters
    // (with the condition that all unbound parameters can be bound to null)
    public override T? Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options)
    {
        if (reader.TokenType != JsonTokenType.StartObject)
        {
            throw new JsonException();
        }

        var candidates = typeToConvert.GetConstructors().Select(ci => new ConstructorMatcher(ci)).ToImmutableList();

        while (reader.Read())
        {
            if (reader.TokenType == JsonTokenType.EndObject)
            {
                var bestCandidate = candidates
                                    .Where(c => !c.HasNonNullableUnmatchedParameters)
                                    .OrderByDescending(c => c.MatchCount)
                                              .ThenBy(c => c.UnmatchedCount)
                                              .FirstOrDefault() ??
                                    throw new NoSuitableConstructorFoundException("<useful exception message>");
                return (T?)bestCandidate.ConstructorInfo.Invoke(bestCandidate.GetInvocationParameters());
            }

            if (reader.TokenType != JsonTokenType.PropertyName)
            {
                throw new JsonException();
            }

            var propertyName = reader.GetString() ?? throw new Exception("<useful exception message>");

            candidates = candidates.Where(c => c.HasParameter(propertyName)).ToImmutableList();
            var possibleTypes = candidates.Select(c => c.GetTypeForParameter(propertyName))
                                          .Distinct()
                                          .ToArray();
            if (possibleTypes.Length > 1)
                throw new AmbiguousConfigurationException("<useful exception message>");

            var propertyType = possibleTypes[0];

            var value = JsonSerializer.Deserialize(ref reader, propertyType, options);

            foreach (var constructorMatcher in candidates)
            {
                constructorMatcher.AddParameterValue(propertyName, propertyType, value);
            }
        }

        throw new JsonException();
    }

    public override void Write(Utf8JsonWriter writer, T value, JsonSerializerOptions options)
    {
        writer.WriteStartObject();
        foreach (var property in typeof(T).GetProperties())
        {
            writer.WritePropertyName(options.PropertyNamingPolicy?.ConvertName(property.Name) ?? property.Name);
            JsonSerializer.Serialize(value, options);
        }
        writer.WriteEndObject();
    }
}

Now for the converter factory the problem we have to address is "to which target types exactly our converter should be applied?"

We'd want to handle all Json objects but there is nothing that clearly ties Json objects with a certain class of CLR types specifically. Json objects can be deserialized to POCO classes but also dictionaries and structs. Conversely collection classes will typically be deserialized from Json arrays. Some Json objects can also be handled by custom converters from the serialization options. I'm quite sure our strategy can be improved as we limited ourselves to what was sufficient in our specific situation.

We chose to apply our converter to all classes that don't implement IEnumerable (the main limitation being we dont handle structs, I guess).

Furthermore, if we find a more specific converter in the serialization options that can convert the target type, we will apply it instead of our converter.

public class CustomObjectConverterFactory: JsonConverterFactory
{
    public override bool CanConvert(Type typeToConvert)
    {
        return typeToConvert.IsClass && !typeof(IEnumerable).IsAssignableFrom(typeToConvert);
    }

    public override JsonConverter? CreateConverter(Type typeToConvert, JsonSerializerOptions options)
    {
        var moreSpecificConverter = options.Converters.FirstOrDefault(c => c != this && c.CanConvert(typeToConvert));
        if (moreSpecificConverter is not null)
            return moreSpecificConverter is JsonConverterFactory moreSpecificFactory
                       ? moreSpecificFactory.CreateConverter(typeToConvert, options)
                       : moreSpecificConverter;
        return (JsonConverter?)Activator.CreateInstance(typeof(CustomObjectConverter<>).MakeGenericType(typeToConvert));
    }
}

Answer 2

I think that the simplest solution is to use classes with nullable values for deserialisation and then transform these into your desired classes that have non-nullable properties.

I think this solution is:

1. very simple
2. easy to understand
3. not taking away compile time checks
4. easy to maintain.

Option 2 below allows arbitrarily complex logic for when the value for a give property is null (throw, log, run some code to get default, get value from configuration).

public class OuterExternal // <-- this is given to the deserialise method
{
  public int? Prop1 { get; }
  ...
  public string? Prop4 { get; }
  ...
}

public class Outer
{
  public int Prop1 { get; }
  ...
  public string Prop4 { get; }
}

You can create Outer objects using a constructor or a factory method.

public class Outer
{
  ...
  // Option 1: Use constructor
  public Outer(OuterExternal external)
    => this(
      prop1: external.prop1 ?? GetSensibleRuntimeProp1Default(),
      ...
      prop4: ...)
  }
}

public class SomeService
{
  // Option 2 Construction done somewhere else:
  private Outer FromExternal(OuterExternal external)
  {
    return new Outer(
      prop1: external.prop1 ?? GetValueForProp1BecauseInputWasNull(someState),
      ...,
      prop4: ...);
  }
}