json.net: how can I deserialize 'similar but different' external json structures to a single class

Question

I'm looking for a way, preferably using JSON.NET (using latest), to deserialize multiple external JSON formats/structures to a single class.

simplified example (the differences are greater than this but the different json's contain similar information):

external 1

{
    "id": 1234,
    "person": {
        "name": "john",
        "surname": "doe"           
    }
}

external 2

{
    "ref": "1234",
    "firstName": "JOHN",
    "lastName": "DOE"
}

internal (this is not real, it's just for show)

{
    "tag": "1234",
    "name1": "John",
    "name2": "Doe"
}

Is there some way/library which perhaps allows you to configure the mapping using a mapping.json file. Preferably one that also allows formatting of the values etc. These are only 2 examples, but we have many more.

Edit: We can tell/hint JSON.NET what source the given JSON is coming from. Therefor we don't have to have a single schema/contract/solution that can handle all different scenarios. We would actually prefer to have a .json mapping/transform config file for every different external json structure (to keep it future proof without the need of having to rebuild everything).

Edit 2: What I've now done is the following based on Pavel Baravik answer is to go through all properties of a 'schema/transformation' JSON. This JSON has the same structure as the final JSON of the object that we want to transform the original external JSON to. If a token is of type 'String' we'll parse that string (supports {{ }} and plain =) and use that as a path to pull values from the original external JSON. In the meantime the final JSON is being constructed, which afterwards will be deserialized to our internal object.
I think we could improve the performance of this code by 'sort-of' compiling it using an Expression tree.

static void Main(string[] args)
{
    var schema = @"
    {
        ""person"": {                    
            ""name"": ""=test.firstName"",
            ""fullName"": ""{{test.firstName}} {{lastName}}"",
            ""surName"":  ""=lastName""
        }
    }";

    var json1 = @"
    {
        ""test"": {
            ""firstName"": ""John""
        },                
        ""lastName"": ""Doe"",
    }";

    Console.WriteLine(Transform(json1, schema).ToString());
    Console.ReadLine();
}

public static JObject Transform(string json, string schema)
{
    var j = JObject.Parse(json);
    var s = JObject.Parse(schema);
    var t = Transform(s, j);

    return t;
}

public static JObject Transform(JObject schema, JObject source)
{
    var target = new JObject();

    foreach (var child in schema.Children())
    {
        var property = child as JProperty;
        if (property != null)
        {
            var schemaToken = property.Value;
            var allowClone = true;

            JToken t = null;
            if (schemaToken.Type == JTokenType.Object)
            {
                t = Transform((JObject) schemaToken, source);
            }
            else if (schemaToken.Type == JTokenType.String)
            {
                allowClone = false;
                t = TransformProperty(source, (JValue)schemaToken);
            }

            if (t != null || allowClone)
            {
                target.Add(property.Name, (t ?? property).DeepClone());
            }
        }
    }

    return target;
}

private static readonly Regex MoustacheRegex = new Regex(@"\{\{[^\}]+\}\}", RegexOptions.Compiled | RegexOptions.CultureInvariant | RegexOptions.Singleline);

private static JToken TransformProperty(JObject source, JValue jstring)
{
    var str = (string)jstring.Value;
    JToken t = null;

    // if string starts with =
    if (str.StartsWith("="))
    {
        t = GetTokenByPath(source, str.Substring(1));
    }
    else
    {
        var allFound = true;

        str = MoustacheRegex.Replace(str, m =>
        {
            var mv = m.Value;
            var mt = GetTokenByPath(source, mv.Substring(2, mv.Length - 4));
            if (mt == null) allFound = false;

            return mt?.ToString() ?? string.Empty;
        });

        if (allFound)
            t = new JValue(str.Trim());
    }

    return t;
}

private static JToken GetTokenByPath(JObject source, string path)
{
    JToken t = null;
    var pathItems = path.Split('.');
    var s = source;

    for (var i = 0; i < pathItems.Length && s != null; ++i, s = t as JObject)
    {
        t = s[pathItems[i]];
    }

    return t;
}

EDIT: (nice JTransform class)

public class JTransform
{
    private InternalJTransform _internal;

    public void Load(string filePath)
    {
        using (var stream = File.OpenRead(filePath))
        using (var reader = new StreamReader(stream))
        {
            Load(new JsonTextReader(reader));
        }
    }

    public void Load(string filePath, Encoding encoding)
    {
        using (var stream = File.OpenRead(filePath))
        using (var reader = new StreamReader(stream, encoding))
        {
            Load(new JsonTextReader(reader));
        }
    }

    public void Load(JsonReader reader)
    {
        _internal = new InternalJTransform(reader);
    }

    public JObject Transform(JsonReader sourceReader)
    {
        return _internal.Transform(sourceReader);
    }

    public JObject Transform(JObject source)
    {
        return _internal.Transform(source);
    }

    public T TransformObject<T>(object obj)
    {
        return _internal.TransformObject<T>(obj);
    }

    public T TransformObject<T>(JObject source, JsonSerializer serializer = null)
    {
        return _internal.TransformObject<T>(source, serializer);
    }

    #region InternalJTransform

    private sealed class InternalJTransform
    {
        private static readonly Regex MoustacheRegex = new Regex(@"\{\{[^\}]+\}\}", RegexOptions.Compiled | RegexOptions.CultureInvariant | RegexOptions.Singleline);

        private JsonSerializer _serializer;
        private JObject _template;
        private bool _ignoreUndefined;

        public InternalJTransform(JsonReader reader)
        {
            var json = JObject.Load(reader);

            _template = json["template"] as JObject;
            _serializer = new JsonSerializer();

            var settings = json["settings"];

            if (settings["camelCase"]?.Value<bool>() ?? false)
                _serializer.ContractResolver = new CamelCasePropertyNamesContractResolver();

            if (settings["ignoreNull"]?.Value<bool>() ?? false)
                _serializer.NullValueHandling = NullValueHandling.Ignore;

            _ignoreUndefined = (settings["ignoreUndefined"]?.Value<bool>() ?? settings["ignoreNull"]?.Value<bool>() ?? false);
        }

        private void Load(JsonReader reader)
        {
            var json = JObject.Load(reader);

            var template = json["template"] as JObject;
            var serializer = new JsonSerializer();

            var settings = json["settings"];

            if (settings["camelCase"]?.Value<bool>() ?? false)
                serializer.ContractResolver = new CamelCasePropertyNamesContractResolver();

            if (settings["ignoreNull"]?.Value<bool>() ?? false)
                serializer.NullValueHandling = NullValueHandling.Ignore;

            _ignoreUndefined = (settings["ignoreNull"]?.Value<bool>() ?? false);
            _serializer = serializer;
            _template = template;
        }

        public JObject Transform(JsonReader sourceReader)
        {
            var obj = JObject.Load(sourceReader);
            return TransformInternal(_template, obj, _serializer);
        }

        public JObject Transform(JObject source)
        {
            return TransformInternal(_template, source, _serializer);
        }

        public T TransformObject<T>(object obj)
        {
            var source = JObject.FromObject(obj);
            var im = TransformInternal(_template, source, _serializer);
            return im.ToObject<T>(_serializer);
        }

        public T TransformObject<T>(JObject source, JsonSerializer serializer = null)
        {
            var obj = TransformInternal(_template, source, _serializer);
            return obj.ToObject<T>(serializer ?? _serializer);
        }

        private JObject TransformInternal(JObject template, JObject source, JsonSerializer serializer)
        {
            var ignoreNull = serializer.NullValueHandling == NullValueHandling.Ignore;
            var target = new JObject();

            foreach (var property in template.Properties())
            {
                var token = property.Value;

                if (token.Type == JTokenType.Object)
                {
                    token = TransformInternal((JObject)token, source, serializer);
                }
                else if (token.Type == JTokenType.String)
                {
                    token = TransformStringToken(source, (JValue)token);

                    // handle undefined, not found, values
                    if (token == null && _ignoreUndefined) continue;
                }

                // handle real null values (this does not include null values set in the template)
                if (token != null && token.Type == JTokenType.Null && ignoreNull) continue;

                target.Add(property.Name, token?.DeepClone());
            }

            return target;
        }

        private JToken TransformStringToken(JObject source, JValue jstring)
        {
            var str = (string)jstring.Value;
            JToken t = null;

            // if string starts with =
            if (str.StartsWith("="))
            {
                t = GetTokenByPath(source, str.Substring(1));
            }
            else
            {
                var allFound = true;

                str = MoustacheRegex.Replace(str, m =>
                {
                    var mv = m.Value;
                    var mt = GetTokenByPath(source, mv.Substring(2, mv.Length - 4));
                    if (mt == null) allFound = false;

                    return mt?.ToString() ?? string.Empty;
                });

                if (allFound)
                    t = new JValue(str.Trim());
            }

            return t;
        }

        private static JToken GetTokenByPath(JObject source, string path)
        {
            JToken t = null;
            var pathItems = path.Split('.');
            var s = source;

            for (var i = 0; i < pathItems.Length && s != null; ++i, s = t as JObject)
            {
                t = s[pathItems[i]];
            }

            return t;
        }
    }

    #endregion
}

DaFi4 · Answer 1 · 2016-02-26T14:42:22.297

0

Take a look at the CustomCreationConverter in JSON.NET http://www.newtonsoft.com/json/help/html/CustomCreationConverter.htm you can make different converters and decide which one to use based on the JSON you have. they could all output the same class

edited Feb 26 '16 at 14:42

answered Feb 26 '16 at 13:36

DaFi4

1,364
9
21

Thank you, going to play around a bit to try and find out whether this is a solution that works for our situation. – alex.pino Feb 26 '16 at 14:51

score 0 · Answer 2 · edited May 23 '17 at 11:45

You can firstly 'flatten' your input structures with use of JsonReader and then map to a single class (adopted from JSON.NET deserialize a specific property).

void Main()
{
    var json0 = @"{
    ""id"": 1234,
    ""person"": {
        ""name"": ""john"",
        ""surname"": ""doe""           
    }";

    var json1 = @"  {
    ""ref"": ""1234"",
    ""firstName"": ""JOHN"",
    ""lastName"": ""DOE""
    }";

    foreach (var j in new []{json0, json1})
    {
        var name = GetFirstInstance<string>(new [] {"person.name", "firstName", "name1"}, j);
        var surname = GetFirstInstance<string> (new [] {"person.surname", "lastName", "name2"}, j);

        new {name, surname}.Dump();
    }
}

public T GetFirstInstance<T>(string[] path, string json)
{
    using (var stringReader = new StringReader(json))
    using (var jsonReader = new JsonTextReader(stringReader))
    {
        while (jsonReader.Read())
        {
            if (jsonReader.TokenType == JsonToken.PropertyName  && path.Contains((string)jsonReader.Path))
            {
                jsonReader.Read();

                var serializer = new JsonSerializer();
                return serializer.Deserialize<T>(jsonReader);
            }
        }
        return default(T);
    }
}

As mentioned above ;-) thank you as well, going to play around a bit to try and find out whether this is a solution that works for our situation. — alex.pino, Feb 26 '16 at 14:52
I've added the code that I've written based on your comments at the top. Don't know whether it's the best solution, but it works. — alex.pino, Feb 29 '16 at 10:07

json.net: how can I deserialize 'similar but different' external json structures to a single class

2 Answers2