Newtonsoft how to deserialize null to default value again after switching to .Net Core

Question

We updated our Asp.Net application from .Net Framework 4.7.2 to .Net 5. Now we have problems with deserialization JSON in the controller methods. In old version we used Newtonsoft.Json. Former if we got a null for a property in JSON for a non nullable type like int the deserializer took the default value, or ignored the null and the error and did not overwrite the property's default value from object creation. But now after the error the whole object is set to null.

{
   "effortType": "1",
   "cwe": null,
   "distanceInKilometers": null,
   "effortDate": "2022-03-22T14:45:00+01:00",
   "effortInHours": 1.0,
   "hours25InHours": null,
   "hours50InHours": null,
   "hours100InHours": null,
   "hours150InHours": null,
   "orderNumber": "006001780872",
   "withCosts": false,
   "isNew": true,
   "isEdited": false,
   "isDeleted": false
}

public class OrderEffortDto
{
    public string EffortType { get; set; }
    public bool Cwe { get; set; }
    public int? DistanceInKilometers { get; set; }
    public DateTimeOffset? EffortDate { get; set; }
    public decimal EffortInHours { get; set; }
    public decimal Hours25InHours { get; set; }
    public decimal Hours50InHours { get; set; }
    public decimal Hours100InHours { get; set; }
    public decimal Hours150InHours { get; set; }
    public string OperationNumber { get; set; }
    public bool IsNew { get; set; }
    public bool IsEdited { get; set; }
    public bool IsDeleted { get; set; }
}

Expected like before would be a OrderEffortDto instance with Cwe = false and all HoursXXInHours = 0

What we get is OrderEffortDto = null

We already tried to use Newtonsoft also in new version, but with the same result. We also configured SerializerSettings.NullValueHandling = NullValueHandling.Ignore. This works for that problem, but than the null values are also ignored for the other direction, for serialization of DTOs into JSON, where the nulls are needed.

Is there a way to get back the old behavior? Right, it would be no problem to fix that in front end to get the right values into the JSON, but our application is big and to determine all the places, where we have to correct that is fault-prone.

Update 1 for those who may have the same problem

I created two simple test projects one ASP.Net WebApi with .Net Framework 4.7.2 and one ASP.Net WebApi with .Net 5, with above JSON and DTO example. I got two similar traces with errors from Newtonsoft and already described results for the DTO in the Controllers. Also the System.Text.Json in .Net 5 gave me a null for the whole DTO.

For API with .Net Framework 4.7.2

2022-03-24T10:50:05.368 Info Started deserializing WebApplication1NetFramework.Data.OrderEffortDto. Path 'effortType', line 2, position 16.
2022-03-24T10:50:05.388 Error Error deserializing WebApplication1NetFramework.Data.OrderEffortDto. Error converting value {null} to type 'System.Boolean'. Path 'cwe', line 3, position 14.
2022-03-24T10:50:05.403 Error Error deserializing WebApplication1NetFramework.Data.OrderEffortDto. Error converting value {null} to type 'System.Decimal'. Path 'hours25InHours', line 7, position 25.
2022-03-24T10:50:05.403 Error Error deserializing WebApplication1NetFramework.Data.OrderEffortDto. Error converting value {null} to type 'System.Decimal'. Path 'hours50InHours', line 8, position 25.
2022-03-24T10:50:05.403 Error Error deserializing WebApplication1NetFramework.Data.OrderEffortDto. Error converting value {null} to type 'System.Decimal'. Path 'hours100InHours', line 9, position 26.
2022-03-24T10:50:05.404 Error Error deserializing WebApplication1NetFramework.Data.OrderEffortDto. Error converting value {null} to type 'System.Decimal'. Path 'hours150InHours', line 10, position 26.
2022-03-24T10:50:05.404 Verbose Could not find member 'orderNumber' on WebApplication1NetFramework.Data.OrderEffortDto. Path 'orderNumber', line 11, position 17.
2022-03-24T10:50:05.405 Verbose Could not find member 'withCosts' on WebApplication1NetFramework.Data.OrderEffortDto. Path 'withCosts', line 12, position 15.
2022-03-24T10:50:05.407 Info Finished deserializing WebApplication1NetFramework.Data.OrderEffortDto. Path '', line 16, position 1.
2022-03-24T10:50:05.407 Verbose Deserialized JSON: 
{
  "effortType": "1",
  "cwe": null,
  "distanceInKilometers": null,
  "effortDate": "2022-03-22T14:45:00+01:00",
  "effortInHours": 1.0,
  "hours25InHours": null,
  "hours50InHours": null,
  "hours100InHours": null,
  "hours150InHours": null,
  "orderNumber": "006001780872",
  "withCosts": false,
  "isNew": true,
  "isEdited": false,
  "isDeleted": false
}

For API with .Net 5

2022-03-24T10:48:19.162 Info Started deserializing WebApplication1NetCore.Data.OrderEffortDto. Path 'effortType', line 2, position 16.
2022-03-24T10:48:19.180 Error Error deserializing WebApplication1NetCore.Data.OrderEffortDto. Error converting value {null} to type 'System.Boolean'. Path 'cwe', line 3, position 14.
2022-03-24T10:48:19.196 Error Error deserializing WebApplication1NetCore.Data.OrderEffortDto. Error converting value {null} to type 'System.Decimal'. Path 'hours25InHours', line 7, position 25.
2022-03-24T10:48:19.196 Error Error deserializing WebApplication1NetCore.Data.OrderEffortDto. Error converting value {null} to type 'System.Decimal'. Path 'hours50InHours', line 8, position 25.
2022-03-24T10:48:19.197 Error Error deserializing WebApplication1NetCore.Data.OrderEffortDto. Error converting value {null} to type 'System.Decimal'. Path 'hours100InHours', line 9, position 26.
2022-03-24T10:48:19.197 Error Error deserializing WebApplication1NetCore.Data.OrderEffortDto. Error converting value {null} to type 'System.Decimal'. Path 'hours150InHours', line 10, position 26.
2022-03-24T10:48:19.197 Verbose Could not find member 'orderNumber' on WebApplication1NetCore.Data.OrderEffortDto. Path 'orderNumber', line 11, position 17.
2022-03-24T10:48:19.197 Verbose Could not find member 'withCosts' on WebApplication1NetCore.Data.OrderEffortDto. Path 'withCosts', line 12, position 15.
2022-03-24T10:48:19.199 Info Finished deserializing WebApplication1NetCore.Data.OrderEffortDto. Path '', line 16, position 1.
2022-03-24T10:48:19.200 Verbose Deserialized JSON: 
{
  "effortType": "1",
  "cwe": null,
  "distanceInKilometers": null,
  "effortDate": "2022-03-22T14:45:00+01:00",
  "effortInHours": 1.0,
  "hours25InHours": null,
  "hours50InHours": null,
  "hours100InHours": null,
  "hours150InHours": null,
  "orderNumber": "006001780872",
  "withCosts": false,
  "isNew": true,
  "isEdited": false,
  "isDeleted": false
}

Thanks go out to @dbc for the comments. I will try it with the convertor in the mentioned post Json.net deserialization null guid case, but also will log the occurrences to fix the root cause.

Update 2

I altered the converter a little and used the "SwaggerGen.TypeExtensions.GetDefaultValue()". So I was able to remove the generic and use one converter for all non-nullable types.

public class NullToDefaultConverter : JsonConverter
{
    public override bool CanConvert(Type objectType)
    {
        var defaultValue = objectType.GetDefaultValue();
        return defaultValue != null;
    }

    public override object ReadJson(JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer)
    {
        var token = JToken.Load(reader);
        if (token.Type == JTokenType.Null)
             // here I will add a logger to get all faulty calls
             return objectType.GetDefaultValue();
        return token.ToObject(objectType); // Deserialize using default serializer
    }

    // Return false I don't want default values to be written as null
    public override bool CanWrite => false;

    public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer)
    {
        throw new NotImplementedException();
    }
}

@das.fliagsi could you try [JsonProperty] on the top of property that can be null and see if it deserialize correctly — Gaurav Chaudhary, Mar 23 '22 at 14:13
Unfortunately does not work. It would work with [JsonProperty(NullValueHandling = NullValueHandling.Ignore)], but than again I have the problem the other direction. — das.flaigsi, Mar 23 '22 at 14:40
In .NET 5 have you switched from Json.NET to System.Text.Json? By the way, I don't think Json.NET will automatically deserialize a `null` value to the default value of a non-nullable value type; you need a converter for it. See e.g. [Json.net deserialization null guid case](https://stackoverflow.com/q/31747712/3744182). — dbc, Mar 23 '22 at 18:24
@dbc Yes I tried it also with System.Text.Json. There I get also a null object for the whole object. The settings for NullValueHandling and DefaultValueHandling are very similar. One benefit of Json.Net is the TraceWriter. And because of this I agree to you that there may not be a conversion. Because this traces shows the errors while deserializing the null values for non-nullable types. But this errors are ignored and the properties are skipped. Therefore since normal creation of the OrderEffortDto object, the non-nullable properties are already filled with the defaults. — das.flaigsi, Mar 24 '22 at 08:12
@das.flaigsi - if Json.NET was always generating errors, but in the past asp.net would ignore them during model binding on a per-property basis but now they cause model binding to fail completely, rather than attempting to restore the old behavior, I would *fix the errors*. Adding `NullToDefaultConverter` from [Json.net deserialization null guid case](https://stackoverflow.com/a/31750851/3744182) for all necessary `T` to `JsonSerializerSettings.Converters` should do the job. In fact I might suggest closing this as a duplicate, agree? — dbc, Mar 24 '22 at 12:40

CajunCoding · Answer 1 · 2022-11-24T02:51:42.243

This is a great question and one that I also encountered recently when migrating a large (error plagued) legacy code base from .NET Framework to .NET Core.

The OP's updates are very helpful but I wanted to share a slightly simplified and higher performing solution that eliminates the dependency on SwaggerGen.TypeExtensions.GetDefaultValue() as well as no Generic dependency; so it can be applied globally as a Converter:

private static readonly NullToDefaultConverter _jsonLegacyCompatibleNullValueConverter = new();

   . . .

var model = JsonConvert.DeserializeObject(content, modelType, _jsonLegacyCompatibleNullValueConverter);

When Json.NET is deserializing the current existing or default value is provided via the object existingValue input parameter which will have the default value for a property or the initial value set by the property initializer. In both cases we likely want to preserve that value and therefore can just return it thereby providing behavior that is more compatible with the legacy .NET Framework Web API behavior (as described in the OP's post).

Notice also that we only have to handle the properties that can not be assigned a null value, so a more performant check can be done there on the Type (adapted from the elegant stack overflow answer here).

/// <summary>
/// This is a Json Converter that restores legacy ASP.Net MVC compatible behavior for handling values that cannot be assigned Null.
/// Historically (before .NET Core) Json values of null would result in errors when setting them into values that cannot be assigned null,
///     however these errors treated as warnings and were skipped, leaving the original default values set on the Model. Now in 
///     Asp .NET Core, these failures result in Exceptions being thrown.
/// While previously the result was that these fields were simply left their default values; either the default of the Value type 
///     or the default set in the Property Initializer.
/// Therefore this Json Converter restores this behavior by assigning the Default value safely without any errors being thrown.
/// </summary>
public class NullToDefaultConverter : JsonConverter
{
    public static bool CanTypeBeAssignedNull(Type type)
        => !type.IsValueType || (Nullable.GetUnderlyingType(type) != null);

    //Only Handle Fields that would fail a Null assignment and requires resolving of a non-null existing/default value for the Type!
    public override bool CanConvert(Type objectType) => !CanTypeBeAssignedNull(objectType);

    public override object ReadJson(JsonReader reader, Type objectType, object existingOrDefaultValue, JsonSerializer serializer)
    {
        var token = JToken.Load(reader);
        return token.Type == JTokenType.Null ? existingOrDefaultValue : token.ToObject(objectType);
    }

    // Return false; we want normal Json.NET behavior when serializing...
    public override bool CanWrite => false;

    public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer) => throw new NotImplementedException();
}

And to further expand on how to restore legacy compatible behavior globally, we implemented a custom Input Formatter as follows:

    public class JsonLegacyCompatibilityInputFormatter : TextInputFormatter
    {
        public bool EnableExceptions { get; set; }

        private static readonly NullToDefaultConverter _jsonLegacyCompatibleNullValueConverter = new();

        public JsonLegacyCompatibilityInputFormatter(bool enableExceptions = true)
        {
            EnableExceptions = enableExceptions;
            //Add Supported Media Types
            SupportedMediaTypes.Add(MediaTypeHeaderValue.Parse("application/json"));
            //Add Supported Encodings
            SupportedEncodings.Add(Encoding.UTF8);
            SupportedEncodings.Add(Encoding.Unicode);
        }

        public override async Task<InputFormatterResult> ReadRequestBodyAsync(InputFormatterContext context, Encoding effectiveEncoding)
        {
            var httpRequest = context.HttpContext.Request;
            var modelType = context.ModelType;
            try
            {
                using var streamReader = new StreamReader(httpRequest.Body, effectiveEncoding);
                var content = await streamReader.ReadToEndAsync().ConfigureAwait(false);

                //NOTE: There is a deviation between .NET Framework and Asp .NET Core handling of null values with properties that cannot be assigned a null value.
                //      Therefore To be compatible with Legacy Web API behavior we  provide the JsonLegacyCompatibleNullValueConverter to ensure
                //          that properties that cannot be assigned null from null Json values retain their original or default value without exceptions being thrown.
                var model = JsonConvert.DeserializeObject(content, modelType, _jsonLegacyCompatibleNullValueConverter);

                return await InputFormatterResult.SuccessAsync(model).ConfigureAwait(false);
            }
            catch (Exception exc)
            {
                var message = $"Error occurred during Model binding de-serialization of [{modelType.Name}].";
                serviceProvider.GetService<ILogger>()?.LogError(exc, message);

                #if DEBUG
                Debug.WriteLine(exc.ConvertExceptionToJson());
                #endif

                if (EnableExceptions)
                    throw new Exception(message, exc);
                else
                    return await InputFormatterResult.FailureAsync().ConfigureAwait(false);
            }
        }
    }

Which is configured for all model binding to use via:

// Configure MVC Services...
builder.Services
    .AddControllersWithViews(options =>
    {
        options.InputFormatters.Insert(0, new JsonLegacyCompatibilityInputFormatter());
    })

EDIT - Optimized version now shared on Nuget:

Since originally sharing this solution I realized that there were several performance impacts and a couple other compatibility items, such as error handling, that needed to be addressed. I've now shared an optimized version , on GitHub & Nuget, that implements a custom JsonBufferedHttpRequestReader and CharArrayPool vs serializing to string with unnecessary allocations. It also enables legacy compatibility for error handling by pushing all errors into the ModelState and preventing those exceptions from being thrown.

This full version is available on Nuget and located here: https://github.com/cajuncoding/AspNetCoreMigrationShims

And can be configured easily just as Newtonsoft is via:

builder.Services
   .AddControllersWithViews(options => {})
   .AddNewtonsfotJson(options => {}) //Original Newtsonsoft Configuration
   .WithNewtonsoftJsonNetFrameworkCompatibility(); //Now updated and re-configured for better Legacy Compatibility

Serge · Answer 2 · 2022-03-23T14:44:45.450

1

just make properties nullable

public class OrderEffortDto
{
    .........
    public bool? Cwe { get; set; }
    public decimal? EffortInHours { get; set; }
    public decimal? Hours25InHours { get; set; }
    public decimal? Hours50InHours { get; set; }
    public decimal? Hours100InHours { get; set; }
    public decimal? Hours150InHours { get; set; }
   
}

or you can add a constructor instead of making nullable. You can only include in the constructor the properties that needs to be changed during deserialization

[Newtonsoft.Json.JsonConstructor]
     public OrderEffortDto(
     bool? cwe ,
     decimal? effortInHours ,
     decimal? hours25InHours ,
     decimal? hours50InHours ,
     decimal? hours100InHours ,
     decimal? hours150InHours )
    {
        Cwe = cwe==null?false: (bool) Cwe;
        EffortInHours = effortInHours==null? 0: (decimal) effortInHours;
       .... and so on
    }

edited Mar 23 '22 at 14:44

answered Mar 23 '22 at 14:25

Serge

40,935
4
18
45

1

No, because it is not nullable for the business logic, at least not for all of them. – das.flaigsi Mar 23 '22 at 14:43
@das.flaigsi See my updated answer – Serge Mar 23 '22 at 14:45
Uff..ok, that will work. But almost the same like fixing it in front end. It will be hard to find all the places and it is fault-prone. But I will keep it in mind. Thank you. – das.flaigsi Mar 23 '22 at 14:58
I had a similar issue as the post, but this is a much simpler solution and I'd rather do things the "new" way – raider33 Aug 17 '22 at 12:27

Newtonsoft how to deserialize null to default value again after switching to .Net Core

2 Answers2

Linked