Compare 2 JSON Files by keys and return only the difference in key values using python

Question

JSON1:

{"test":{"name":"test","bucket":{"name":"."},"format":{"JSON":{"path":""}},"state":{"id":"01658204657296583104","schema":{"InputSchema":{"RecordColumns":[{"Mapping":"0","Name":"column0","SqlType":"STRING"},{"Mapping":"1","Name":"column1","SqlType":"STRING"},{"Mapping":"2","Name":"column2","SqlType":"STRING"},{"Mapping":"3","Name":"column3","SqlType":"STRING"},{"Mapping":"4","Name":"column4","SqlType":"STRING"},{"Mapping":"5","Name":"column5","SqlType":"STRING"},{"Mapping":"6","Name":"column6","SqlType":"STRING"},{"Mapping":"7","Name":"time_column7","SqlType":"PERIOD"},{"Mapping":"8","Name":"column8","SqlType":"STRING"},{"Mapping":"9","Name":"column9","SqlType":"STRING"},{"Mapping":"10","Name":"column10","SqlType":"NUMBER"},{"Mapping":"11","Name":"column11","SqlType":"STRING"},{"Mapping":"12","Name":"column12","SqlType":"STRING"},{"Mapping":"13","Name":"column13","SqlType":"STRING"},{"Mapping":"14","Name":"column14","SqlType":"NUMBER"},{"Mapping":"15","Name":"time_column15","SqlType":"PERIOD"},{"Mapping":"16","Name":"column16","SqlType":"NUMBER"},{"Mapping":"17","Name":"column17","SqlType":"STRING"},{"Mapping":"18","Name":"column18","SqlType":"STRING"}],"RecordEncoding":"UTF-8","RecordFormat":{"MappingParameters":{"JSONMappingParameters":{"RecordRowPath":"$"}},"RecordFormatType":"JSON"}}},"progress":[637,637],"sources":{"./allure.json":{"loc":"./allure.json","etag":"None","id":""}},"events":[]}}}

JSON2:

{"test":{"name":"test","bucket":{"name":"."},"format":{"JSON":{"path":""}},"state":{"id":"01658204657296583104","schema":{"InputSchema":{"RecordColumns":[{"Mapping":"0","Name":"column0","SqlType":"STRING"},{"Mapping":"1","Name":"column1","SqlType":"STRING"},{"Mapping":"2","Name":"column2","SqlType":"STRING"},{"Mapping":"3","Name":"column3","SqlType":"STRING"},{"Mapping":"4","Name":"column4","SqlType":"STRING"},{"Mapping":"5","Name":"column5","SqlType":"STRING"},{"Mapping":"6","Name":"column6","SqlType":"STRING"},{"Mapping":"7","Name":"time_column7","SqlType":"PERIOD"},{"Mapping":"8","Name":"column8","SqlType":"STRING"},{"Mapping":"9","Name":"column9","SqlType":"STRING"},{"Mapping":"10","Name":"column10","SqlType":"STRING"},{"Mapping":"11","Name":"column11","SqlType":"STRING"},{"Mapping":"12","Name":"column12","SqlType":"STRING"},{"Mapping":"13","Name":"column13","SqlType":"STRING"},{"Mapping":"14","Name":"column14","SqlType":"NUMBER"},{"Mapping":"15","Name":"time_column15","SqlType":"PERIOD"},{"Mapping":"16","Name":"column16","SqlType":"NUMBER"},{"Mapping":"17","Name":"column17","SqlType":"STRING"},{"Mapping":"18","Name":"column18","SqlType":"STRING"}],"RecordEncoding":"UTF-8","RecordFormat":{"MappingParameters":{"JSONMappingParameters":{"RecordRowPath":"$"}},"RecordFormatType":"JSON"}}},"progress":[637,637],"sources":{"./allure.json":{"loc":"./allure.json","etag":"None","id":"01658204657296583104"}},"events":[]}}}

Can anyone help me out this question? I want a python script that will compare both JSON files and return only the differed key values.

Your data shows embedded dictionaries. Do you want to compare keys at every level or only at a given level of embedding ? — sciroccorics, Apr 17 '18 at 06:58
@Jonathan: There is a missing `':'` after `test` at the very beginning — sciroccorics, Apr 17 '18 at 07:01

score 0 · Answer 1 · answered Apr 17 '18 at 07:32

This simple code uses a recursive function to extract all values in all embedded list/dict. Then it compares both lists obtained by parsing json1 and json2 and detects all different values at similar list indices:

json1 = {"test":{"name":"test","bucket":{"name":"."},"format":{"JSON":{"path":""}},"state":{"id":"01658204657296583104","schema":{"InputSchema":{"RecordColumns":[{"Mapping":"0","Name":"column0","SqlType":"STRING"},{"Mapping":"1","Name":"column1","SqlType":"STRING"},{"Mapping":"2","Name":"column2","SqlType":"STRING"},{"Mapping":"3","Name":"column3","SqlType":"STRING"},{"Mapping":"4","Name":"column4","SqlType":"STRING"},{"Mapping":"5","Name":"column5","SqlType":"STRING"},{"Mapping":"6","Name":"column6","SqlType":"STRING"},{"Mapping":"7","Name":"time_column7","SqlType":"PERIOD"},{"Mapping":"8","Name":"column8","SqlType":"STRING"},{"Mapping":"9","Name":"column9","SqlType":"STRING"},{"Mapping":"10","Name":"column10","SqlType":"NUMBER"},{"Mapping":"11","Name":"column11","SqlType":"STRING"},{"Mapping":"12","Name":"column12","SqlType":"STRING"},{"Mapping":"13","Name":"column13","SqlType":"STRING"},{"Mapping":"14","Name":"column14","SqlType":"NUMBER"},{"Mapping":"15","Name":"time_column15","SqlType":"PERIOD"},{"Mapping":"16","Name":"column16","SqlType":"NUMBER"},{"Mapping":"17","Name":"column17","SqlType":"STRING"},{"Mapping":"18","Name":"column18","SqlType":"STRING"}],"RecordEncoding":"UTF-8","RecordFormat":{"MappingParameters":{"JSONMappingParameters":{"RecordRowPath":"$"}},"RecordFormatType":"JSON"}}},"progress":[637,637],"sources":{"./allure.json":{"loc":"./allure.json","etag":"None","id":""}},"events":[]}}}
json2 = {"test":{"name":"test","bucket":{"name":"."},"format":{"JSON":{"path":""}},"state":{"id":"01658204657296583104","schema":{"InputSchema":{"RecordColumns":[{"Mapping":"0","Name":"column0","SqlType":"STRING"},{"Mapping":"1","Name":"column1","SqlType":"STRING"},{"Mapping":"2","Name":"column2","SqlType":"STRING"},{"Mapping":"3","Name":"column3","SqlType":"STRING"},{"Mapping":"4","Name":"column4","SqlType":"STRING"},{"Mapping":"5","Name":"column5","SqlType":"STRING"},{"Mapping":"6","Name":"column6","SqlType":"STRING"},{"Mapping":"7","Name":"time_column7","SqlType":"PERIOD"},{"Mapping":"8","Name":"column8","SqlType":"STRING"},{"Mapping":"9","Name":"column9","SqlType":"STRING"},{"Mapping":"10","Name":"column10","SqlType":"STRING"},{"Mapping":"11","Name":"column11","SqlType":"STRING"},{"Mapping":"12","Name":"column12","SqlType":"STRING"},{"Mapping":"13","Name":"column13","SqlType":"STRING"},{"Mapping":"14","Name":"column14","SqlType":"NUMBER"},{"Mapping":"15","Name":"time_column15","SqlType":"PERIOD"},{"Mapping":"16","Name":"column16","SqlType":"NUMBER"},{"Mapping":"17","Name":"column17","SqlType":"STRING"},{"Mapping":"18","Name":"column18","SqlType":"STRING"}],"RecordEncoding":"UTF-8","RecordFormat":{"MappingParameters":{"JSONMappingParameters":{"RecordRowPath":"$"}},"RecordFormatType":"JSON"}}},"progress":[637,637],"sources":{"./allure.json":{"loc":"./allure.json","etag":"None","id":"01658204657296583104"}},"events":[]}}}

def get_values(json, lst):
  if isinstance(json, list):
    for item in json: get_values(item, lst)
  elif isinstance(json, dict):
    for item in json.values(): get_values(item, lst)
  else: lst.append(json)

list1 = []; get_values(json1, list1)
list2 = []; get_values(json2, list2)

diff = [(n, x, y) for n,(x,y) in enumerate(zip(list1, list2)) if x != y]
print(diff)

Result:

[(36, 'NUMBER', 'STRING'), (68, '', '01658204657296583104')]

I don't want this code in list. By using list if we change the order, then it will provide me a different output. Without using list I want the script and also those two json files should get for a different path, not in the same code. — Sabarish Manikandan, Apr 18 '18 at 12:57
If you only extract values that are different in both files, how could you know where these different values are located in your data structure? My answer provides you an index that allows you to easily locate the differences. If this is not what you want, you should provide an example of desired output in your initial post. Nobody knows what you have in your mind if you don't write it down. — sciroccorics, Apr 18 '18 at 15:31

Compare 2 JSON Files by keys and return only the difference in key values using python

1 Answers1