System.Text.Json and Dynamically Parsing polymorphic objects

Question

I don't believe I am wrapping my head around how to properly use JsonConverter for polymorphism in parsing json results.

In my scenario, I am targeting Git Policy Configurations in TFS. A policy configuration:


"value": [
{
        "createdBy": {
            "displayName": "username",
            "url": "url",
            "id": "id",
            "uniqueName": "user",
            "imageUrl": "url"
        },
        "createdDate": "2020-03-21T18:17:24.3240783Z",
        "isEnabled": true,
        "isBlocking": true,
        "isDeleted": false,
        "settings": {
            "minimumApproverCount": 1,
            "creatorVoteCounts": false,
            "allowDownvotes": false,
            "resetOnSourcePush": true,
            "scope": [{
                    "refName": "refs/heads/master",
                    "matchKind": "Exact",
                    "repositoryId": "id"
                }
            ]
        },
        "_links": {
            "self": {
                "href": "url"
            },
            "policyType": {
                "href": "url"
            }
        },
        "revision": 1,
        "id": 974,
        "url": "url",
        "type": {
            "id": "id",
            "url": "url",
            "displayName": "Minimum number of reviewers"
        },
{...}]

More settings examples: Require a Merge Strategy

"settings": {
        "useSquashMerge": true,
        "scope": [
            {
                "refName": "refs/heads/master",
                "matchKind": "Exact",
                "repositoryId": "id"
            }
        ]
    }

Required Reviewers

    "settings": {
        "requiredReviewerIds": [
            "id"
        ],
        "scope": [
            {
                "refName": "refs/heads/master",
                "matchKind": "Exact",
                "repositoryId": "id"
            }
        ]
    }

In the json snippet above, the settings object is different based on the type of configuration.

What is the best approach to writing a converter than can dynamically serialize/deserialize the settings object? I've read a couple of articles regarding this and can't quite wrap my head around it.

This is how I am currently deserializing all of my API results, so far they have been simple result sets.

async Task<List<T>> ParseResults<T>( HttpResponseMessage result, string parameter )
{
    List<T> results = new List<T>();

    if ( result.IsSuccessStatusCode )
    {
        using var stream = await result.Content.ReadAsStreamAsync();
        JsonDocument doc = JsonDocument.Parse( stream );
        JsonElement collection = doc.RootElement.GetProperty( parameter ).Clone();

        foreach ( var item in collection.EnumerateArray() )
        {
            results.Add( JsonSerializer.Deserialize<T>( item.ToString() ) );
        }
    }

    return results;
}

My integration test.

PolicyConfiguration is the type I am trying to deserialize to.

[Test]
public async Task Get_TestMasterBranchPolicyConfigurations()
{
    HttpResponseMessage result = await GetResult( $"{_collection}/ProductionBuildTesting/_apis/policy/configurations?api-version=4.1" );

    List<PolicyConfiguration> configurations = await ParseResults<PolicyConfiguration>( result, "value" );
    Assert.AreEqual( 16, configurations.Count );
    JsonPrint( configurations );
}

My current classes for this parsing situation

public class CreatedBy
{
    [JsonPropertyName( "displayName" )]
    public string DisplayName { get; set; }
    [JsonPropertyName( "url" )]
    public string Url { get; set; }
    [JsonPropertyName( "id" )]
    public Guid Id { get; set; }
    [JsonPropertyName( "uniqueName" )]
    public string UniqueName { get; set; }
    [JsonPropertyName( "imageUrl" )]
    public string ImageUrl { get; set; }
}

public class PolicyConfigurationScope
{
    [JsonPropertyName( "refName" )]
    public string RefName { get; set; }
    [JsonPropertyName( "matchKind" )]
    public string MatchKind { get; set; }
    [JsonPropertyName( "repositoryId" )]
    public Guid RepositoryId { get; set; }
}

public class PolicyConfigurationSettings_MinimumNumberOfReviewers
{
    [JsonPropertyName( "minimumApproverCount" )]
    public int MinimumApproverCount { get; set; }
    [JsonPropertyName( "creatorVoteCounts" )]
    public bool CreatorVoteCounts { get; set; }
    [JsonPropertyName( "allowDownvotes" )]
    public bool AllowDownvotes { get; set; }
    [JsonPropertyName( "resetOnSourcePush" )]
    public bool ResetOnSourcePush { get; set; }
    [JsonPropertyName( "scope" )]
    public List<PolicyConfigurationScope> Scope { get; set; }
}

public class PolicyConfigurationType
{
    [JsonPropertyName( "id" )]
    public Guid Id { get; set; }
    [JsonPropertyName( "url" )]
    public string Url { get; set; }
    [JsonPropertyName( "displayName" )]
    public string DisplayName { get; set; }
}

public class PolicyConfiguration
{
    [JsonPropertyName( "createdBy" )]
    public CreatedBy CreatedBy { get; set; }
    [JsonPropertyName( "createdDate" )]
    public DateTime CreatedDate { get; set; }
    [JsonPropertyName( "isEnabled" )]
    public bool IsEnabled { get; set; }
    [JsonPropertyName( "isBlocking" )]
    public bool IsBlocking { get; set; }
    [JsonPropertyName( "isDeleted" )]
    public bool IsDeleted { get; set; }
    //[JsonPropertyName( "settings" )]
    //public PolicyConfigurationSettings_MinimumNumberOfReviewersSettings Settings { get; set; }
    [JsonPropertyName( "revision" )]
    public int Revision { get; set; }
    [JsonPropertyName( "id" )]
    public int Id { get; set; }
    [JsonPropertyName( "url" )]
    public string Url { get; set; }
    [JsonPropertyName( "type" )]
    public PolicyConfigurationType Type { get; set; }
}

`settings` is a single token in your sample, why do you serialize to the `List`? Are you parsing the `scope` values? Which `parameter` did you pass, which typeparams are used as `T`? — Pavel Anikhouski, Mar 21 '20 at 19:33
@PavelAnikhouski That is my method to serialize the entire json result. In this case the result is a count of 4 and the parent object is an array of ```value``` types. ```settings``` is a child object of value, and is dynamic. I can post my object types if that helps? Essentially this is a ```PolicyConfiguration```, and the property is ```PolicyConfigurationSettings```. — MattB, Mar 21 '20 at 19:35
@PavelAnikhouski I have updated my question with the classes used for deserializing. Settings is commented out and everything works, when I plug in the class for the listed setting, it attempts to parse Settings as that for every configuration, from here is where I am unsure where to go with a Converter. — MattB, Mar 21 '20 at 19:45
--Edits: I believe I have everything in the original question now to address my current setup. — MattB, Mar 21 '20 at 19:50
Thanks, the question is more clear now. Do you need an exact type for deserialization, have you considered to use a dictionary of keys and values? — Pavel Anikhouski, Mar 21 '20 at 20:09
@PavelAnikhouski I'm honestly not sure what approach to take. This is my first time working with System.Text.Json, and even when using Newtonsoft, I hadn't encountered a complex object like this that I needed to work with. So any advice would be great. I can provide more ```settings``` examples if you would like. I appreciate your assistance! — MattB, Mar 21 '20 at 20:11

score 8 · Accepted Answer · answered Mar 26 '20 at 22:05

I ended up solving my issue in slightly the same way I had seen a previous article using a discriminator. Since I do not control the API feeds, I do not have a discriminator to drive off of, so I am relying on the properties of the Json object.

Need to create a Converter:

public class PolicyConfigurationSettingsConverter : JsonConverter<PolicyConfigurationSettings>
{
    public override PolicyConfigurationSettings Read( ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options )
    {
        JsonDocument doc;
        JsonDocument.TryParseValue( ref reader, out doc );

        if ( doc.RootElement.TryGetProperty( "minimumApproverCount", out _ ) )
            return JsonSerializer.Deserialize<MinimumNumberOfReviewers>( doc.RootElement.ToString(), options );
        if ( doc.RootElement.TryGetProperty( "useSquashMerge", out _ ) )
            return JsonSerializer.Deserialize<RequireAMergeStrategy>( doc.RootElement.ToString(), options );
        if ( doc.RootElement.TryGetProperty( "scope", out _ ) )
            return JsonSerializer.Deserialize<PolicyConfigurationSettingsScope>( doc.RootElement.ToString(), options );

        return null;
    }

    public override void Write( Utf8JsonWriter writer, [DisallowNull] PolicyConfigurationSettings value, JsonSerializerOptions options )
    {
        if ( value.GetType() == typeof( MinimumNumberOfReviewers ) )
            JsonSerializer.Serialize( writer, ( MinimumNumberOfReviewers )value, options );
        if ( value.GetType() == typeof( RequireAMergeStrategy ) )
            JsonSerializer.Serialize( writer, ( RequireAMergeStrategy )value, options );
        if ( value.GetType() == typeof( PolicyConfigurationSettingsScope ) )
            JsonSerializer.Serialize( writer, ( PolicyConfigurationSettingsScope )value, options );
    }
}

Then need to create a JsonSerializerOptions object to add the Converter

public static JsonSerializerOptions PolicyConfigurationSettingsSerializerOptions()
{
    var serializeOptions = new JsonSerializerOptions();
    serializeOptions.Converters.Add( new PolicyConfigurationSettingsConverter() );
    return serializeOptions;
}

Pass the options into your Serializer/Deserializer statement.

Below is the PolicyConfigurationSettings class

public abstract class PolicyConfigurationSettings
{
    [JsonPropertyName( "scope" )]
    public List<PolicyConfigurationScope> Scope { get; set; }
}

public class MinimumNumberOfReviewers : PolicyConfigurationSettings
{
    [JsonPropertyName( "minimumApproverCount" )]
    public int MinimumApproverCount { get; set; }
    [JsonPropertyName( "creatorVoteCounts" )]
    public bool CreatorVoteCounts { get; set; }
    [JsonPropertyName( "allowDownvotes" )]
    public bool AllowDownvotes { get; set; }
    [JsonPropertyName( "resetOnSourcePush" )]
    public bool ResetOnSourcePush { get; set; }
}

public class RequireAMergeStrategy : PolicyConfigurationSettings
{
    [JsonPropertyName( "useSquashMerge" )]
    public bool UseSquashMerge { get; set; }
}

public class PolicyConfigurationSettingsScope : PolicyConfigurationSettings { }

`JsonDocument` implements `IDisposable` and in fact must be disposed because *This class utilizes resources from pooled memory to minimize the impact of the garbage collector (GC) in high-usage scenarios. Failure to properly dispose this object will result in the memory not being returned to the pool, which will increase GC impact across various parts of the framework* according to the [docs](https://learn.microsoft.com/en-us/dotnet/api/system.text.json.jsondocument?view=netcore-3.1#remarks). — dbc, Jul 23 '20 at 15:43
Also, you may get better performance re-serializing your `JsonDocument` to a byte array rather than a string for final deserialization. See [System.Text.Json.JsonElement ToObject workaround](https://stackoverflow.com/a/59047063/3744182). — dbc, Jul 26 '20 at 14:42
Thank you @dbc, I will make these changes to my code and update this answer! — MattB, Aug 07 '20 at 13:36

score 2 · Answer 2 · answered Dec 09 '20 at 08:44

In net 5.0 with System.Text.Json.JsonSerializer, what works for a class like this:

public class A
{
    public B Data { get; set; }
}
public class B
{
    public long Count { get; set; }
}

is using:

System.Text.Json.JsonSerializer.Deserialize<A>("{{\"data\":{\"count\":10}}}", new JsonSerializerOptions { PropertyNameCaseInsensitive = true, IncludeFields = true })

which is weird that is not the default.

score 2 · Answer 3 · answered Feb 24 '21 at 15:19

I solved this with a more generic approach, that falls somewhere between the way NewtonSoft Json and the .NET Json work. Using a custom converter, I serialize any polymorphic class, using a type identifier similar to the Newtonsoft approach, but to mitigate the possible security risk you can chose to allow only internal types or types from a specific assembly.

using System.Text.Json;
using System.Text.Json.Serialization;
using System.Diagnostics.CodeAnalysis;
using System.Diagnostics;
using System.Collections.ObjectModel;

public class JsonConverterEx<T> : System.Text.Json.Serialization.JsonConverter<T>
{
    private bool _internalOnly = true;
    private string _assembly = String.Empty;

    public JsonConverterEx()
    {
        this._assembly = this.GetType().Assembly.FullName;
    }

    public JsonConverterEx(bool bInternalOnly, string assemblyName)
    {
        _internalOnly = bInternalOnly;
        _assembly = assemblyName;
    }

    public override bool CanConvert(Type typeToConvert)
    {
        Type t = typeof(T);

        if(typeToConvert == t)
            return true;

        return false;
    }

    public override T Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options)
    {
        validateToken(reader, JsonTokenType.StartObject);

        reader.Read();      // Move to property name
        validateToken(reader, JsonTokenType.PropertyName);

        var typeKey = reader.GetString();

        reader.Read();      // Move to start of object (stored in this property)
        validateToken(reader, JsonTokenType.StartObject);

        if(!_internalOnly)
        {
            typeKey += ", " + _assembly;
        }

        Type t = Type.GetType(typeKey);
        if(t != null)
        {
            T o = (T)JsonSerializer.Deserialize(ref reader, t, options);
            reader.Read(); // Move past end of item object

            return o;
        }
        else
        {
            throw new JsonException($"Unknown type '{typeKey}'");
        }

        // Helper function for validating where you are in the JSON
        void validateToken(Utf8JsonReader reader, JsonTokenType tokenType)
        {
            if(reader.TokenType != tokenType)
                throw new JsonException($"Invalid token: Was expecting a '{tokenType}' token but received a '{reader.TokenType}' token");
        }
    }

    public override void Write(Utf8JsonWriter writer, [DisallowNull] T value, JsonSerializerOptions options)
    {
        var itemType = value.GetType();

        writer.WriteStartObject();
        writer.WritePropertyName(itemType.FullName);

        // pass on to default serializer
        JsonSerializer.Serialize(writer, value, itemType, options);

        writer.WriteEndObject();
    }
}

How to use it:

        JsonSerializerOptions op = new JsonSerializerOptions()
        {
            // your usual options here
        };
        op.Converters.Add(new JsonConverterEx<MyExternalClass>(false, "MyAssembly"));
        op.Converters.Add(new JsonConverterEx<MyInternalClass>());

        string s = System.Text.Json.JsonSerializer.Serialize(myobj, op);

        MyInternalClass c = System.Text.Json.JsonSerializer.Deserialize<MyInternalClass>(s, op);

Antony Kriukov · Answer 4 · 2022-06-26T09:59:42.997

0

Alternatively, a more flexible design for serialization

public class PolymorphicConverter<T> : JsonConverter<T> where T : class
{
    public override T Read(
        ref Utf8JsonReader reader, 
        Type typeToConvert, 
        JsonSerializerOptions options)
    {
        throw new NotImplementedException();
    }

    public override void Write(
        Utf8JsonWriter writer, 
        [DisallowNull] T value, 
        JsonSerializerOptions options)
    {
        JsonSerializer.Serialize(writer, value, value.GetType(), options);
    }
}

next, you can turn on your fantasy and customize deserialization

Do not forget:

options.JsonSerializerOptions.Converters.Add(new PolymorphicConverter<IFucker>());

edited Jun 26 '22 at 09:59

answered Jun 20 '22 at 07:50

Antony Kriukov

1
1

"Will of fantasy"? – Jeremy Caney Jun 23 '22 at 18:45
yes, sorry for wrong context, edited – Antony Kriukov Jun 26 '22 at 10:01

System.Text.Json and Dynamically Parsing polymorphic objects

4 Answers4

Linked

Related