Finding common value pair fields from numerous json in Python

Question

I have list of JSON files. Now I intend to find all the common value pairs from all these JSON and copy it to different JSON. Also the common value pairs should be removed from all JSON's.

lets say I have a.json, b.json, c.json ... z.json

Now the common label value pair in all of them is

"Town" : "New York"

then, this common element should be moved to a new JSON file called common.json and also the element should be removed from all the JSON files.

An eg json file would look like:

{
     "RepetitionTime": 2, 
     "EchoTime": 0,
     "MagneticFieldStrength": 3, 
     "SequenceVariant": "SK",
     "MRAcquisitionType": "2D",
     "FlipAngle": 90,
     "ScanOptions": "FS",
     "SliceTiming": [[0.0025000000000000022], [0.5], [-0.030000000000000027], [0.46625], [-0.06374999999999997], [0.43375000000999997], [-0.09624999999999995], [0.40000000001], [-0.12999999999], [0.36750000001], [-0.16249999998999998], [0.333750000005], [-0.19624999999500004], [0.301250000005], [-0.228749999995], [0.26749999999999996], [-0.26249999999500007], [0.235], [-0.29500000000000004], [0.20124999999999998], [-0.32875], [0.16875000001], [-0.36124999999999996], [0.13500000001], [-0.39499999999], [0.10250000000999998], [-0.42749999999], [0.06875000000499998], [-0.46124999999500005], [0.036250000005000005]],
     "SequenceName": "epfid2d1_64",
     "ManufacturerModelName": "TrioTim",
      "TaskName": "dis",
    "ScanningSequence": "EP",
      "Manufacturer": "SIEMENS"
}

I way i am thinking is too complex. I thought to take each line and of first json file and check with all other jsons.

There should be something easy and efficient. any pointers?

json can hold arbitrary nested/structured combinations of json-objects and json-arrays. No one can help you if you don't give us an idea of what your json files are like. Also, *this is not a code writing service*. You need to demonstrate your own attempts, and how they have failed, and any other research you may have done. — juanpa.arrivillaga, Feb 28 '17 at 22:32
can yous show us the structure of the JSON. the same json files expected to be the same structure. That is can you expect "Town" to be in all files? — parsethis, Feb 28 '17 at 22:32

score 2 · Accepted Answer · edited May 23 '17 at 12:09

To compare all files in one time, you can also use Sets to compare all key-values at once using &

>>> import json

>>> json_dict1 = json.loads('{"a":1, "b":2}')
>>> json_dict2 = json.loads('{"a":1, "b":4, "c":5}')
>>> json_dict3 = json.loads('{"a":1, "b":2, "c":5}')

>>> a = set(json_dict1.items())
>>> b = set(json_dict2.items())
>>> c = set(json_dict3.items())
>>> a & b & c
{('a', 1)}

Note that you can also do other operations with Sets, here an example from the doc:

>>> a = set('abracadabra')
>>> b = set('alacazam')
>>> a                                  # unique letters in a
{'a', 'r', 'b', 'c', 'd'}
>>> a - b                              # letters in a but not in b
{'r', 'd', 'b'}
>>> a | b                              # letters in either a or b
{'a', 'c', 'r', 'd', 'b', 'm', 'z', 'l'}
>>> a & b                              # letters in both a and b
{'a', 'c'}
>>> a ^ b                              # letters in a or b but not both
{'r', 'd', 'b', 'm', 'z', 'l'}

EDIT

Finally, I asked my own so question based on the same problem as you (almost).

Here is the best response

If you are too lazy to click on the link, here is an overview:

>>> list_dict = [json_dict1, json_dict2, json_dict3]
>>> {k: v
     for k, v in list_dict[0].items()
     if all(k in d and d[k] == v
            for d in list_dict[1:])}
{'a': 1}

My anwser doesn't work if the dictionnary have `list` because it will throw a `TypeError: unhashable type: 'list'`. I'll remove this answer tommorow please do not downvote meanwhile ^^' — Kruupös, Mar 01 '17 at 00:10
as per json example shared above, one or more value pairs are lists. For eg: `"SliceTiming"`.. in that case does it work? — learnningprogramming, Mar 01 '17 at 17:07
Unfortunatly no it does not. you can find a way around it by manually removing `"SliceTiming"` or casting it to into `string`. But it is far from perfect! If I manage to solve this problem I'll let you know. Until then I think @Haifeng's solution is the best — Kruupös, Mar 01 '17 at 18:26

score 1 · Answer 2 · answered Feb 28 '17 at 22:45

Since you didn't provide your exact JSON sample, I assume it is just regular json as '{"key":"value"}'.

convert json string to dictionary:

import json
json_dict = json.loads('{"a":1, "b":2}')  # converts json string to dictionary

now assume we have two converted dictionaries:

>>> dict1= {"a":1,"b":2}
>>> dict2= {"a":1,"b":3}

comparing two dictionaries and finding the common key-value pairs(similarly for the diff k-v pairs), I am using python3:

>>> {k:v  for k, v in dict1.items()  for k1,v1 in dict2.items() if k ==k1 and v==v1}
{'a': 1}

My post showed you the idea how to solve your issue, it might have edge issues for your specific JSON lines, you can modify it and fit your needs. Hope it helps

@downvoter please leave a comment and i will improve my answer — Haifeng Zhang, Mar 01 '17 at 00:56
here the issue is i can load json file with just json.loads('pathtofile'). I have to open json file as someobject and then open. — learnningprogramming, Mar 02 '17 at 00:38

Finding common value pair fields from numerous json in Python

2 Answers2