How to deserialize JSON into flat, Map-like structure?

Question

Have in mind that the JSON structure is not known before hand i.e. it is completely arbitrary, we only know that it is JSON format.

For example,

The following JSON

{
   "Port":
   {
       "@alias": "defaultHttp",
       "Enabled": "true",
       "Number": "10092",
       "Protocol": "http",
       "KeepAliveTimeout": "20000",
       "ThreadPool":
       {
           "@enabled": "false",
           "Max": "150",
           "ThreadPriority": "5"
       },
       "ExtendedProperties":
       {
           "Property":
           [                         
               {
                   "@name": "connectionTimeout",
                   "$": "20000"
               }
           ]
       }
   }
}

Should be deserialized into Map-like structure having keys like (not all of the above included for brevity):

port[0].alias
port[0].enabled
port[0].extendedProperties.connectionTimeout
port[0].threadPool.max

I am looking into Jackson currently, so there we have:

TypeReference<HashMap<String, Object>> typeRef = new TypeReference<HashMap<String, Object>>() {};
Map<String, String> o = objectMapper.readValue(jsonString, typeRef);

However, the resulting Map instance is basically a Map of nested Maps:

{Port={@alias=diagnostics, Enabled=false, Type=DIAGNOSTIC, Number=10033, Protocol=JDWP, ExtendedProperties={Property={@name=suspend, $=n}}}}

While I need flat Map with flatten keys using "dot notation", like the above.

I would rather not implement this myself, although at the moment I don't see any other way...

Jackson (or any other JSON library) can convert the JSON to a map of maps. Going the extra mile is not trivial and the example syntax you're showing could never be generated at runtime in java. You can achieve something similar to what you need using the [Typesafe Config library](https://github.com/typesafehub/config). — Giovanni Botta, Dec 03 '13 at 16:00
OK, so I did Config parseString = ConfigFactory.parseString(portJsonString); Which toString() is something like: Config(SimpleConfigObject({"Port":{"Enabled":"false","Number":"10033","Type":"DIAGNOSTIC","@alias":"diagnostics","ExtendedProperties":{"Property":{"@name":"suspend","$":"n"}},"Protocol":"JDWP"}})) But I am not sure how to flatten this via the Typesafe Config lib? — Svilen, Dec 03 '13 at 16:18

score 52 · Accepted Answer · edited Mar 17 '16 at 00:32

You can do this to traverse the tree and keep track of how deep you are to figure out dot notation property names:

import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.node.ArrayNode;
import com.fasterxml.jackson.databind.node.ObjectNode;
import com.fasterxml.jackson.databind.node.ValueNode;
import java.io.IOException;
import java.util.HashMap;
import java.util.Iterator;
import java.util.Map;
import org.junit.Test;

public class FlattenJson {
  String json = "{\n" +
      "   \"Port\":\n" +
      "   {\n" +
      "       \"@alias\": \"defaultHttp\",\n" +
      "       \"Enabled\": \"true\",\n" +
      "       \"Number\": \"10092\",\n" +
      "       \"Protocol\": \"http\",\n" +
      "       \"KeepAliveTimeout\": \"20000\",\n" +
      "       \"ThreadPool\":\n" +
      "       {\n" +
      "           \"@enabled\": \"false\",\n" +
      "           \"Max\": \"150\",\n" +
      "           \"ThreadPriority\": \"5\"\n" +
      "       },\n" +
      "       \"ExtendedProperties\":\n" +
      "       {\n" +
      "           \"Property\":\n" +
      "           [                         \n" +
      "               {\n" +
      "                   \"@name\": \"connectionTimeout\",\n" +
      "                   \"$\": \"20000\"\n" +
      "               }\n" +
      "           ]\n" +
      "       }\n" +
      "   }\n" +
      "}";

  @Test
  public void testCreatingKeyValues() {
    Map<String, String> map = new HashMap<String, String>();
    try {
      addKeys("", new ObjectMapper().readTree(json), map);
    } catch (IOException e) {
      e.printStackTrace();
    }
    System.out.println(map);
  }

  private void addKeys(String currentPath, JsonNode jsonNode, Map<String, String> map) {
    if (jsonNode.isObject()) {
      ObjectNode objectNode = (ObjectNode) jsonNode;
      Iterator<Map.Entry<String, JsonNode>> iter = objectNode.fields();
      String pathPrefix = currentPath.isEmpty() ? "" : currentPath + ".";

      while (iter.hasNext()) {
        Map.Entry<String, JsonNode> entry = iter.next();
        addKeys(pathPrefix + entry.getKey(), entry.getValue(), map);
      }
    } else if (jsonNode.isArray()) {
      ArrayNode arrayNode = (ArrayNode) jsonNode;
      for (int i = 0; i < arrayNode.size(); i++) {
        addKeys(currentPath + "[" + i + "]", arrayNode.get(i), map);
      }
    } else if (jsonNode.isValueNode()) {
      ValueNode valueNode = (ValueNode) jsonNode;
      map.put(currentPath, valueNode.asText());
    }
  }
}

It produces the following map:

Port.ThreadPool.Max=150, 
Port.ThreadPool.@enabled=false, 
Port.Number=10092, 
Port.ExtendedProperties.Property[0].@name=connectionTimeout, 
Port.ThreadPool.ThreadPriority=5, 
Port.Protocol=http, 
Port.KeepAliveTimeout=20000, 
Port.ExtendedProperties.Property[0].$=20000, 
Port.@alias=defaultHttp, 
Port.Enabled=true

It should be easy enough to strip out @ and $ in the property names, although you could end up with collisions in key names since you said the JSON was arbitrary.

At the end of the day I went with a custom solution very similar to this one therefore marking this as accepted answer. Thanks! — Svilen, Apr 03 '16 at 07:19

score 42 · Answer 2 · answered Aug 05 '15 at 07:21

42

How about using the json-flattener. https://github.com/wnameless/json-flattener

BTW, I am the author of this lib.

String flattenedJson = JsonFlattener.flatten(yourJson);
Map<String, Object> flattenedJsonMap = JsonFlattener.flattenAsMap(yourJson);

// Result:
{
    "Port.@alias":"defaultHttp",
    "Port.Enabled":"true",
    "Port.Number":"10092",
    "Port.Protocol":"http",
    "Port.KeepAliveTimeout":"20000",
    "Port.ThreadPool.@enabled":"false",
    "Port.ThreadPool.Max":"150",
    "Port.ThreadPool.ThreadPriority":"5",
    "Port.ExtendedProperties.Property[0].@name":"connectionTimeout",
    "Port.ExtendedProperties.Property[0].$":"20000"
}

answered Aug 05 '15 at 07:21

wnameless

491
5
2

really good work. hope u continue to support it and is production ready. good if u can also post performance and test coverage metrics on ur github repo. are u still actively developing it ? – Ashish Thukral Mar 01 '17 at 02:18
1

can I unflatt map into json ? – PDS Jul 06 '17 at 13:25
1

Works relatively good, BUT this is converting my Integer props to BigDecimal what is very ugly and not useful for me, because it forces additional routines to re-convert types from BigDecimal. – erik.aortiz Sep 26 '19 at 00:51
It's a great library. – Anmol Jain Aug 10 '22 at 07:35

score 8 · Answer 3 · answered Jan 25 '15 at 10:04

how about that:

import java.util.ArrayList;
import java.util.HashMap;
import java.util.Map;
import java.util.Map.Entry;
import com.google.gson.Gson;

/**
 * NOT FOR CONCURENT USE
*/
@SuppressWarnings("unchecked")
public class JsonParser{

Gson gson=new Gson();
Map<String, String> flatmap = new HashMap<String, String>();

public Map<String, String> parse(String value) {        
    iterableCrawl("", null, (gson.fromJson(value, flatmap.getClass())).entrySet());     
    return flatmap; 
}

private <T> void iterableCrawl(String prefix, String suffix, Iterable<T> iterable) {
    int key = 0;
    for (T t : iterable) {
        if (suffix!=null)
            crawl(t, prefix+(key++)+suffix);
        else
            crawl(((Entry<String, Object>) t).getValue(), prefix+((Entry<String, Object>) t).getKey());
    }
}

private void crawl(Object object, String key) {
    if (object instanceof ArrayList)
        iterableCrawl(key+"[", "]", (ArrayList<Object>)object);
    else if (object instanceof Map)
        iterableCrawl(key+".", null, ((Map<String, Object>)object).entrySet());
    else
        flatmap.put(key, object.toString());
}
}

score 4 · Answer 4 · answered Apr 18 '16 at 20:52

org.springframework.integration.transformer.ObjectToMapTransformer from Spring Integration produces desired result. By default it has shouldFlattenKeys property set to true and produces flat maps (no nesting, value is always simple type). When shouldFlattenKeys=false it produces nested maps

ObjectToMapTransformer is meant to be used as part of integration flow, but it is perfectly fine to use it in stand-alone way. You need to construct org.springframework.messaging.Message with payload of transformation input. transform method returns org.springframework.messaging.Message object with payload that is Map

import org.springframework.integration.transformer.ObjectToMapTransformer;
import org.springframework.messaging.Message;
import org.springframework.messaging.support.GenericMessage;

Message message = new GenericMessage(value);
 ObjectToMapTransformer transformer = new ObjectToMapTransformer();
        transformer.setShouldFlattenKeys(true);
        Map<String,Object> payload = (Map<String, Object>) transformer
                .transform(message)
                .getPayload();

Side note: It is probably overkill to add Spring Integration to the classpath just to use single class, but you may check implementation of this class and write similar solution on your own. Nested map is produced by Jackson (org.springframework.integration.support.json.JsonObjectMapper#fromJson(payload, Map.class)), then mapis travered recursively, flattening all values that are collections.

The last line can be much simpler as: `Map payload = transformer.doTransform(message);` — Vikas Prasad, Jul 16 '17 at 03:10

score 3 · Answer 5 · answered Feb 07 '21 at 14:01

I also had to solve a similar problem in my project and found out that springframework.vault has a method flatten() to do the same. Below is a sample code.


    //Json string to Map<String, Object>

    String data = "Your json as string"
    final ObjectMapper mapper = new ObjectMapper();
    final MapType type = mapper.getTypeFactory().constructMapType(
                Map.class, String.class, Object.class);
    final Map<String, Object> map = mapper.readValue(data, type);

    //Using springframework.vault flatten method

    Map<String, String> keyMap = JsonMapFlattener.flattenToStringMap(map);

    //Input

    {"key": {"nested": 1}, "another.key": ["one", "two"] }

    //Output

      key.nested=1
      another.key[0]=one
      another.key[1]=two

Remember to add the dependency

    <dependency>
        <groupId>org.springframework.vault</groupId>
        <artifactId>spring-vault-core</artifactId>
        <version>2.1.1.RELEASE</version>
    </dependency>

For more info, refer https://docs.spring.io/spring-vault/docs/current/api/org/springframework/vault/support/JsonMapFlattener.html

score 1 · Answer 6 · answered Dec 03 '13 at 16:28

1

You can achieve something like that using the Typesafe Config Library as in the following example:

import com.typesafe.config.*;
import java.util.Map;
public class TypesafeConfigExample {
  public static void main(String[] args) {
    Config cfg = ConfigFactory.parseString(
      "   \"Port\":\n" +
      "   {\n" +
      "       \"@alias\": \"defaultHttp\",\n" +
      "       \"Enabled\": \"true\",\n" +
      "       \"Number\": \"10092\",\n" +
      "       \"Protocol\": \"http\",\n" +
      "       \"KeepAliveTimeout\": \"20000\",\n" +
      "       \"ThreadPool\":\n" +
      "       {\n" +
      "           \"@enabled\": \"false\",\n" +
      "           \"Max\": \"150\",\n" +
      "           \"ThreadPriority\": \"5\"\n" +
      "       },\n" +
      "       \"ExtendedProperties\":\n" +
      "       {\n" +
      "           \"Property\":\n" +
      "           [                         \n" +
      "               {\n" +
      "                   \"@name\": \"connectionTimeout\",\n" +
      "                   \"$\": \"20000\"\n" +
      "               }\n" +
      "           ]\n" +
      "       }\n" +
      "   }\n" +
      "}");

    // each key has a similar form to what you need
    for (Map.Entry<String, ConfigValue> e : cfg.entrySet()) {
      System.out.println(e);
    }
  }
}

answered Dec 03 '13 at 16:28

Giovanni Botta

9,626
5
51
94

Thank you, Giovanni. While this is indeed a step closer to what I need it is still not enough. For example it produces stuff like: Port."@alias"=ConfigString("defaultHttp") and Port.ExtendedProperties.Property=SimpleConfigList([{"@name":"jaasRealm","$":"PlatformManagement"}]) - clearly not flattened property set. So, If I want to use that I would need to additionaly parse the output of the Typesafe lib.... I guess I'll just use my own solution and will post it here when it is done. – Svilen Dec 11 '13 at 09:09
Well what you're trying to do is a very "custom" thing, because you have annotation style field names that represent something different. As far as I know the concept is not a "standard" JSON one or otherwise, thus you will need to do some leg work here. I hope the Typesafe library helps though. – Giovanni Botta Dec 11 '13 at 14:59
1

You just need to do some more processing if you want a different format, instead of just calling toString. Each entry has a path expression which can be broken up with `ConfigUtil.splitPath` if you want, and the ConfigValue can be converted to a plain Java value with `value.unwrapped()`. Given the split-up path you could reassemble it using your desired syntax. – Havoc P Jan 11 '14 at 00:55
@Svilen Did you solve it? Could you post your solution as I have the same problem now? – Battle_Slug Sep 10 '14 at 09:54
1

I ended up implementing it myself using Jackson APIs. Basically use Jackson to parse the json to a map (of maps and lists) and then traverse this recursively and build a flat map. Similar to siledh answer – Svilen Sep 23 '14 at 12:34

score -1 · Answer 7 · answered Dec 03 '13 at 15:47

-1

If you know the structure beforehand, you can define a Java class and use gson to parse JSON into an instance of that class:

YourClass obj = gson.fromJson(json, YourClass.class);

If not, then I'm not sure what you're trying to do. You obviously can't define a class on-the-fly so accessing the parsed JSON using dot-notation is out of the question.

Unless you want something like:

Map<String, String> parsed = magicParse(json);
parsed["Port.ThreadPool.max"]; // returns 150

If so, then traversing your map of maps and building a "flattened" map doesn't seem too much of a problem.

Or is it something else?

answered Dec 03 '13 at 15:47

siledh

3,268
2
16
29

The model is not known beforehand. The idea is to pass this flatten JSON to a comparsion service, which, well, to compare it against another json that is flatten in the same way. The problem is that the service expects specific format i.e. this flattened thing I am trying to achieve... I was thinking that this should be not that uncommon problem, so a solution should be already available by some lib. – Svilen Dec 03 '13 at 15:59
@Svilen Why can't you just compare JSON strings themselves? – siledh Dec 03 '13 at 16:40
siledh, the reason is that I would like to have easy way to change just a single property/leaf value after a diff has been made. This would me much more difficult to do having plain a text diff... – Svilen Dec 10 '13 at 12:06
1

@Svilen Why do you need to flatten it then? Map seems sufficient to me. – siledh Dec 11 '13 at 08:34

How to deserialize JSON into flat, Map-like structure?

7 Answers7

Linked