Comparing JSON with arrays

Question

I am trying to compare 2 JSON files, they have arrays with duplicated values.

My first JSON Object has an array like this:

"categories": [
            "May",
            "Apr",
            "Mar"
          ]

My second JSON object has an array like this:

"categories": [
            "May",
            "May",
            "Apr",
            "Apr",
            "Mar",
            "Mar"
          ]

I am comparing the JSON using flat maps that can be found in this link comparing JSONs using guava

Here is part of my code:

 private String smartJSONsCompare(JSONObject leftJson, JSONObject rightJson) {

    String result = "</br>";
    Gson gson = new Gson();
    Type type = new TypeToken<Map<String, Object>>(){}.getType();
    Map<String, Object> leftMap = gson.fromJson(leftJson.toString(), type);
    Map<String, Object> rightMap = gson.fromJson(rightJson.toString(), type);
    Map<String, Object> leftFlatMap = FlatMapUtil.flatten(leftMap);
    Map<String, Object> rightFlatMap = FlatMapUtil.flatten(rightMap);
    MapDifference<String, Object> difference = Maps.difference(leftFlatMap, rightFlatMap);
    StringBuilder SB = new StringBuilder("</br>");
    SB.append("Entries only on LEFT: </br>");
    difference.entriesOnlyOnLeft().forEach((key, value) -> SB.append(key + ": " + value + "</br>"));
    SB.append("Entries only on RIGHT: </br>");
    difference.entriesOnlyOnRight().forEach((key, value) -> SB.append(key + ": " + value + "</br>"));
    SB.append("Entries full difference : </br>");
    difference.entriesDiffering().forEach((key, value) -> SB.append(key + ": " + value + "</br>"));
    result = SB.toString();
    return result;
}

I wish to be able to present the difference in a more "smart" way. In other words: showing all the objects / arrays in the JONSs that don't match. What is missing or what was added to the compared JSON.

For the "categories" array my code returns a message that their is a mismatch, but doesn't state the difference in an elegant way.

What can I do?

Answer 1

I have change a bit in your solution to get the wanted result.

I would do my difference check in List, therefore I will create method to change JSON to list of strings based on your code:

private static List<String> jsonToList(String json){
    List<String> list = new ArrayList<>();
    Gson gson = new Gson();
    Type type = new TypeToken<Map<String, Object>>(){}.getType();
    Map<String, Object> jsonMap = gson.fromJson(json, type);
    Map<String, Object> flatten = FlatMapUtil.flatten(jsonMap);
    flatten.forEach((k, v) -> list.add(v.toString()));
    return list;
}

---

Update
When I answered the question I did things a bit fast, the jsonToList was based on your code. As it is right now it is over complicated to what you are asking for. I have therefore made much lighter version using the following method in stead:

private static List<String> jsonToList(String json) {
    JSONObject response = new JSONObject(json);
    List<String> list = new ArrayList<>();
    JSONArray jsonArray = response.getJSONArray("categories");
    if (jsonArray != null) {
        for (int i = 0; i < jsonArray.length(); i++) {
            list.add(jsonArray.get(i).toString());
        }
    }
    return list;
}

That said, now you have two choices and it is up to you to find out which one fits best to your needs and take it from here.

End of Update

---

for this example I have made 3 test examples

String main = "{\"categories\":[\"May\",\"Apr\",\"Mar\"]}";
String json1 = "{\"categories\":[\"May\",\"May\",\"Apr\",\"Apr\",\"Mar\",\"Mar\"]}";
String json2 = "{\"categories\":[\"May\",\"Apr\",\"Apr\",\"Mar\",\"Mar\",\"Mar\"]}";
String json3 = "{\"categories\":[\"May\",\"Apr\",\"Mar\",\"Mar\"]}";

in my second step I will create a

List<String> mainList = jsonToList(main);
List<String> list1 = jsonToList(json1);

so far so good. Now I make a method to take the extra difference of the 2 list, that mean as you requested in your comments, we take only all values that are duplicated more than once and return them in list. In this method I used hashmap only count duplicates and than take the all that is repeated more than 1 time:

private static List<String> diffList(List<String> mainList, List<String> secondList){
    List<String> list = new ArrayList<String>();
    Map<String, Integer> wordCount = new HashMap<>();
    for(String word: secondList) {
        if(mainList.contains(word)) {
            Integer count = wordCount.get(word);
            wordCount.put(word, (count == null) ? 1 : count + 1);
            if(wordCount.get(word) > 1){
                list.add(word);
            }
        }
    }
    return list;
}

Finally I would test all cases, for instance for list1:

List<String> diff1 = diffList(mainList, list1);
for (String s : diff1) {
    System.out.println(s);
}

The output will be

May
Apr
Mar

for list2

Apr
Mar
Mar

And for list3

Mar

Now I will separate view method from the your method and create some thing like, just to make my code more clear and easy to work with:

private static String viewResult(List<String> list1, List<String> list2, List<String> duplicate){
    String result;
    StringBuilder SB = new StringBuilder("</br>");
    SB.append("Entries only on LEFT: </br>");
    list1.forEach(e -> SB.append(e + "</br>"));
    SB.append("Entries only on RIGHT: </br>");
    list2.forEach(e -> SB.append(e + "</br>"));
    SB.append("Entries full difference : </br>");
    duplicate.forEach(e -> SB.append(e + "</br>"));
    result = SB.toString();
    return result;
}

So if we put all this code together I will be some thing like this, and the following code is to demonstrate how things works, but from here you can take it to the next level in your code:

public static void main(String[] args) {
    String main = "{\"categories\":[\"May\",\"Apr\",\"Mar\"]}";
    String json1 = "{\"categories\":[\"May\",\"May\",\"Apr\",\"Apr\",\"Mar\",\"Mar\"]}";
    String json2 = "{\"categories\":[\"May\",\"Apr\",\"Apr\",\"Mar\",\"Mar\",\"Mar\"]}";
    String json3 = "{\"categories\":[\"May\",\"Apr\",\"Mar\",\"Mar\"]}";

    List<String> mainList = jsonToList(main);

    List<String> list1 = jsonToList(json1);
    List<String> diff1 = diffList(mainList, list1);
    for (String s : diff1) {
        System.out.println(s);
    }

    String view = viewResult(mainList, list1, diff1);
}

private static List<String> jsonToList(String json){
    List<String> list = new ArrayList<String>();
    Gson gson = new Gson();
    Type type = new TypeToken<Map<String, Object>>(){}.getType();
    Map<String, Object> jsonMap = gson.fromJson(json, type);
    Map<String, Object> flatten = FlatMapUtil.flatten(jsonMap);
    flatten.forEach((k, v) -> list.add(v.toString()));
    return list;
}

private static List<String> diffList(List<String> mainList, List<String> secondList){
    List<String> list = new ArrayList<String>();
    Map<String, Integer> wordCount = new HashMap<>();
    for(String word: secondList) {
        if(mainList.contains(word)) {
            Integer count = wordCount.get(word);
            wordCount.put(word, (count == null) ? 1 : count + 1);
            if(wordCount.get(word) > 1){
                list.add(word);
            }
        }
    }
    return list;
}

private static String viewResult(List<String> list1, List<String> list2, List<String> duplicate){
    String result;
    StringBuilder SB = new StringBuilder("</br>");
    SB.append("Entries only on LEFT: </br>");
    list1.forEach(e -> SB.append(e + "</br>"));
    SB.append("Entries only on RIGHT: </br>");
    list2.forEach(e -> SB.append(e + "</br>"));
    SB.append("Entries full difference : </br>");
    duplicate.forEach(e -> SB.append(e + "</br>"));
    result = SB.toString();
    return result;
}

Answer 2

If you want something more generic with a good diff you could utilize AssertJ here. Its usually used for Testing, but the diff looks really good and you can also use it in normal code.

Example:

Expecting:
  <["Mai", "Apr", "Mar"]>
to contain exactly in any order:
  <["May", "Apr", "Mar", "Mar"]>
elements not found:
  <["May", "Mar"]>
and elements not expected:
  <["Mai"]>

Can be created by:

[...]
import org.assertj.core.api.Assertions;

public class JsonTest {
    final static String arr = " [\n"+
            "            \"Mai\",\n"+
            "            \"Apr\",\n"+
            "            \"Mar\"\n"+
            "          ]";
    final static String arr2 = " [\n"+
            "            \"May\",\n"+
            "            \"Apr\",\n"+
            "            \"Mar\",\n"+
            "            \"Mar\"\n"+
            "          ]";

    public static void main(String[] args){
        System.out.println(smartJSONsCompare(arr,arr2));
    }

    private static String smartJSONsCompare(String leftJson, String rightJson) {
        Gson gson = new Gson();
        Type type = new TypeToken<List<String>>(){}.getType();
        List<String> left = gson.fromJson(leftJson, type);
        List<String> right = gson.fromJson(rightJson, type);
        try{
            Assertions.assertThat(left).containsExactlyInAnyOrderElementsOf(right);
        }catch(AssertionError  ae){
            return ae.getMessage();
        }
        return "Matched";
    }
  }

I added the dependencies in gradle with:

dependencies {
    compile("org.assertj:assertj-core:3.11.1")
}

Answer 3

If you want to create a patch between your two JSON Objects have a look at json-patch.

import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.github.fge.jsonpatch.diff.JsonDiff;
import java.io.IOException;

public class JsonPatchTest {
    public static void main(String[] args) throws IOException {
        String jsonFirst = "{\"categories\":[\"May\",\"Apr\",\"Mar\"]}";
        String jsonSecond = "{\"categories\":[\"May\",\"May\",\"Apr\",\"Apr\",\"Mar\",\"Mar\"]}";

        ObjectMapper mapper = new ObjectMapper();
        JsonNode jsonNodeFirst = mapper.readTree(jsonFirst);
        JsonNode jsonNodeSecond = mapper.readTree(jsonSecond);

        JsonNode patchNode = JsonDiff.asJson(jsonNodeFirst, jsonNodeSecond);

        System.out.println(mapper.writerWithDefaultPrettyPrinter().writeValueAsString(patchNode));
    }
}

Would produce the following output for your scenario:

[ {
  "op" : "replace",
  "path" : "/categories/1",
  "value" : "May"
}, {
  "op" : "replace",
  "path" : "/categories/2",
  "value" : "Apr"
}, {
  "op" : "add",
  "path" : "/categories/-",
  "value" : "Apr"
}, {
  "op" : "add",
  "path" : "/categories/-",
  "value" : "Mar"
}, {
  "op" : "add",
  "path" : "/categories/-",
  "value" : "Mar"
} ]

Answer 4

I believe you should handle json arrays on your own in order to present their difference in a more "smart" way. Here is a library which contains CollectionUtils class with disjunction method.

MapDifference<String, Object> difference = Maps.difference(leftMap, rightMap);
difference.entriesDiffering().forEach((key, value) -> {
  Object left = value.leftValue();
  Object right = value.rightValue();
  if (left instanceof Iterable && right instanceof Iterable) {
    Collection<?> diff = CollectionUtils.disjunction((Iterable<?>) right, (Iterable<?>) left);
    System.out.println(key + " -> " + diff);
  }
  ...
});

Answer 5

This code work for me (2 years ago) on production.

public class App {
private final Gson GSON = new GsonBuilder().create();

public boolean isDifference(final String path, Map<String, Object> oldData, Map<String, Object> newData) {
    MapDifference<String, Object> difference = Maps.difference(oldData, newData);
    difference.entriesOnlyOnLeft().forEach((key, value) -> {
        publishChange(Action.REMOVE, path, key, value);
    });
    difference.entriesOnlyOnRight().forEach((key, value) -> {
        publishChange(Action.ADD, path, key, value);
    });
    difference.entriesDiffering().forEach((key, value) -> {
        if (value.rightValue() instanceof Map && value.leftValue() instanceof Map) {
            if (!path.isEmpty()) {
                key = path.concat("-").concat(key);
            }
            isDifference(key, (Map) value.leftValue(), (Map) value.rightValue());
        } else {
            publishChange(Action.MODIFY, path, key, value);
        }
    });
    return !difference.areEqual();
}

public void publishChange(Action action, String path, String key, Object value) {
    if (value instanceof MapDifference.ValueDifference) {
        value = ((MapDifference.ValueDifference) value).rightValue();
    }
    JsonElement jsonValue = GSON.toJsonTree(value);
    String event = createEvent(action, path, key, jsonValue);
    System.out.println("Differrence: " + event);
}

public String createEvent(Action action, String paths, String key, JsonElement value) {
    JsonObject root = new JsonObject();
    JsonArray arrPaths = new JsonArray();
    for (String path : paths.split("-")) {
        arrPaths.add(path);
    }
    root.addProperty("action", action.toString());
    root.add("paths", arrPaths);
    JsonObject data = new JsonObject();
    data.addProperty("key", key);
    data.add("value", value);
    root.add("data", data);
    return root.toString();
}

public static enum Action {
    ADD, REMOVE, MODIFY
}}

Test/ Example:

public class AppTest {
@Test
public void testAppHasAGreeting() {
    App classUnderTest = new App();

    Gson gson = new GsonBuilder()
            .setPrettyPrinting()
            .create();
    // JsonOld: {"a":1,"b":1,"c":true,"array":[1,2,3],"object":{"arrayKey":["a","b","c","d"]}}
    String jsonOld = "{\"a\":1,\"b\":1,\"c\":true,\"array\":[1,2,3],\"object\":{\"arrayKey\":[\"a\",\"b\",\"c\",\"d\"]}}";
    // JsonNew: {"a":2,"b":1,"array":[1,2,3,2],"another":{"d":false,"e":["a","b","c"]},"object":{"booleanKey":true,"arrayKey":["a","b","c"]}}
    String jsonNew = "{\"a\":2,\"b\":1,\"array\":[1,2,3,2],\"another\":{\"d\":false,\"e\":[\"a\",\"b\",\"c\"]},\"object\":{\"booleanKey\":true,\"arrayKey\":[\"a\",\"b\",\"c\"]}}";

    Type mapType = new TypeToken<Map<String, Object>>() {
    }.getType();
    Map<String, Object> jsonOldAsMap = gson.fromJson(jsonOld, mapType);
    Map<String, Object> jsonNewAsMap = gson.fromJson(jsonNew, mapType);
    System.out.println("Old Json: " + gson.toJson(jsonOldAsMap));
    System.out.println("New Json: " + gson.toJson(jsonNewAsMap));
    System.out.println("========== Result ==========");
    // When
    boolean diff = classUnderTest.isDifference("", jsonOldAsMap, jsonNewAsMap);
    // Then
    assertTrue(diff);
}}

The result will print like this:

Differrence: {"action":"REMOVE","paths":[""],"data":{"key":"c","value":true}}

Differrence: {"action":"ADD","paths":[""],"data":{"key":"another","value":{"d":false,"e":["a","b","c"]}}}

Differrence: {"action":"MODIFY","paths":[""],"data":{"key":"a","value":2.0}}

Differrence: {"action":"MODIFY","paths":[""],"data":{"key":"array","value":[1.0,2.0,3.0,2.0]}}

Differrence: {"action":"ADD","paths":["object"],"data":{"key":"booleanKey","value":true}}

Differrence: {"action":"MODIFY","paths":["object"],"data":{"key":"arrayKey","value":["a","b","c"]}}

The code available here: https://github.com/liemle3893/compare-json