Python merge nested list of dictionary to single list of dictionary

Question

I want to merge a nested list of dictionaries to single list of a dictionary in python 2.6, Example data - Here is given only two iterations of thousands of iterations.

INPUTJSON=[
 {'EXCEPTIONS': 
            [
              {'LASTOCCURED': '2018-03-12 12:11:23', 'COUNT': 25, 'NAME': 'CLFRW0134W'}, 
              {'LASTOCCURED': '2018-03-12 12:11:42', 'COUNT': 10, 'NAME': 'SRV0145GH'}
            ], 
  'JVM_NAME': 'TestiingAWS01', 'GCCOUNT': 10},

 {'EXCEPTIONS': 
           [
              {'LASTOCCURED': '2018-03-13 12:14:23', 'COUNT': 25, 'NAME': 'CLFRW0134W'}, 
              {'LASTOCCURED': '2018-03-18 12:55:23', 'COUNT': 10, 'NAME': 'SRV0145GH'}
           ], 
  'JVM_NAME': 'QAAWS02', 'GCCOUNT': 10}
]

EXPECTED RESULT:

[
 {'JVM_NAME':'TestiingAWS01','GCCOUNT':10, 'LASTOCCURED': '2018-03-12 12:11:23', 'COUNT': 25, 'NAME': 'CLFRW0134W'}, {'LASTOCCURED': '2018-03-12 12:11:42', 'COUNT': 10, 'NAME': 'SRV0145GH', 'JVM_NAME':'TestiingAWS01','GCCOUNT':10},
 {'JVM_NAME': 'QAAWS02', 'GCCOUNT': 10, 'LASTOCCURED': '2018-03-13 12:14:23', 'COUNT': 25, 'NAME': 'CLFRW0134W'}, {'LASTOCCURED': '2018-03-18 12:55:23', 'COUNT': 10, 'NAME': 'SRV0145GH', 'JVM_NAME': 'QAAWS02', 'GCCOUNT': 10}
]

Requesting experts to help me to achieve this so that I could process the final data to sqlite easily.

UPDATE: Thanks, all experts for prompt solutions, consolidating from all answers here I've got a working code for my data by avoiding hardcode "keys" (as there would be 40 keys on each iteration) on python 2.6.

def merge_two_dicts(x, y):
    """Given two dicts, merge them into a new dict as a shallow copy."""
    z = x.copy()
    z.update(y)
    return z

resultlist =[]
for i,v in enumerate(INPUTJSON):
    EXCEPTIONS = v["EXCEPTIONS"]
    del v["EXCEPTIONS"]
    for j,val in enumerate(EXCEPTIONS):
        resultlist.append(merge_two_dicts(EXCEPTIONS[j],INPUTJSON[i]))

print resultlist

Can it be compiled in comprehension list using lambda?

score 2 · Answer 1 · answered Mar 19 '18 at 01:29

Here is one way.

lst = [{**{'JVM_NAME': i['JVM_NAME'], 'GCCOUNT': i['GCCOUNT'], **w}} \
       for i in INPUTJSON for w in i['EXCEPTIONS']]

For the ** syntax, see How to merge two dictionaries in a single expression?

Result

[{'COUNT': 25,
  'GCCOUNT': 10,
  'JVM_NAME': 'TestiingAWS01',
  'LASTOCCURED': '2018-03-12 12:11:23',
  'NAME': 'CLFRW0134W'},
 {'COUNT': 10,
  'GCCOUNT': 10,
  'JVM_NAME': 'TestiingAWS01',
  'LASTOCCURED': '2018-03-12 12:11:42',
  'NAME': 'SRV0145GH'},
 {'COUNT': 25,
  'GCCOUNT': 10,
  'JVM_NAME': 'QAAWS02',
  'LASTOCCURED': '2018-03-13 12:14:23',
  'NAME': 'CLFRW0134W'},
 {'COUNT': 10,
  'GCCOUNT': 10,
  'JVM_NAME': 'QAAWS02',
  'LASTOCCURED': '2018-03-18 12:55:23',
  'NAME': 'SRV0145GH'}]

Thanks Sir for the working solution considering example data, but I am avoiding to hardcode all the 40 keys from production data, so manipulated the code :) — Rahul Trivedi, Mar 19 '18 at 02:41

Ajax1234 · Answer 2 · 2018-03-19T01:33:01.220

You can try this:

INPUTJSON=[
   {'EXCEPTIONS': 
        [
          {'LASTOCCURED': '2018-03-12 12:11:23', 'COUNT': 25, 'NAME': 
   'CLFRW0134W'}, 
          {'LASTOCCURED': '2018-03-12 12:11:42', 'COUNT': 10, 'NAME': 
  'SRV0145GH'}
        ], 
   'JVM_NAME': 'TestiingAWS01', 'GCCOUNT': 10},

  {'EXCEPTIONS': 
       [
          {'LASTOCCURED': '2018-03-13 12:14:23', 'COUNT': 25, 'NAME': 'CLFRW0134W'}, 
          {'LASTOCCURED': '2018-03-18 12:55:23', 'COUNT': 10, 'NAME': 'SRV0145GH'}
       ], 
    'JVM_NAME': 'QAAWS02', 'GCCOUNT': 10}
]
new_result = [i for b in [[dict([('JVM_NAME', i['JVM_NAME']), ('GCCOUNT', i['GCCOUNT'])]+b.items()) for b in i['EXCEPTIONS']] for i in INPUTJSON] for i in b]

Output:

[{'LASTOCCURED': '2018-03-12 12:11:23', 'JVM_NAME': 'TestiingAWS01', 'GCCOUNT': 10, 'COUNT': 25, 'NAME': 'CLFRW0134W'}, {'LASTOCCURED': '2018-03-12 12:11:42', 'JVM_NAME': 'TestiingAWS01', 'GCCOUNT': 10, 'COUNT': 10, 'NAME': 'SRV0145GH'}, {'LASTOCCURED': '2018-03-13 12:14:23', 'JVM_NAME': 'QAAWS02', 'GCCOUNT': 10, 'COUNT': 25, 'NAME': 'CLFRW0134W'}, {'LASTOCCURED': '2018-03-18 12:55:23', 'JVM_NAME': 'QAAWS02', 'GCCOUNT': 10, 'COUNT': 10, 'NAME': 'SRV0145GH'}]

Note, however, that this problem is simpler in Python3 when utilizing unpacking:

final_result = [i for b in [[{**{a:b for a, b in i.items() if a != 'EXCEPTIONS'}, **c} for c in i['EXCEPTIONS']] for i in INPUTJSON] for i in b]

Output:

[{'LASTOCCURED': '2018-03-12 12:11:23', 'JVM_NAME': 'TestiingAWS01', 'GCCOUNT': 10, 'COUNT': 25, 'NAME': 'CLFRW0134W'}, {'LASTOCCURED': '2018-03-12 12:11:42', 'JVM_NAME': 'TestiingAWS01', 'GCCOUNT': 10, 'COUNT': 10, 'NAME': 'SRV0145GH'}, {'LASTOCCURED': '2018-03-13 12:14:23', 'JVM_NAME': 'QAAWS02', 'GCCOUNT': 10, 'COUNT': 25, 'NAME': 'CLFRW0134W'}, {'LASTOCCURED': '2018-03-18 12:55:23', 'JVM_NAME': 'QAAWS02', 'GCCOUNT': 10, 'COUNT': 10, 'NAME': 'SRV0145GH'}]

Thanks for the solution, however as I cant hardcode all the keys from prod data, so I have created a required code derived from all the answers here, looking for comprehension list :) — Rahul Trivedi, Mar 19 '18 at 02:51

score 1 · Accepted Answer · answered Mar 19 '18 at 01:38

1

data = list()
for item in INPUTJSON:
    EXCEPTIONS = item["EXCEPTIONS"]
    del item["EXCEPTIONS"]
    for ex in EXCEPTIONS:
        tmp = dict()
        tmp.update(ex)    
        tmp.update(item)
        data.append(tmp)
print data

answered Mar 19 '18 at 01:38

guichao

198
1
7

Thanks, @Guichao, your solution is straight to result and I adopted your logic to code according to my data :) – Rahul Trivedi Mar 19 '18 at 02:44

score 1 · Answer 4 · answered Mar 19 '18 at 06:27

You can try:

final_one=[]
for i in data:
    final=[]
    temp={}
    for m,n in i.items():
        if not isinstance(n,list):
            temp[m]=n
        else:
            final+=n
    for h in final:
        h.update(temp)
    final_one+=final
print(final_one)

output:

[{'COUNT': 25, 'NAME': 'CLFRW0134W', 'LASTOCCURED': '2018-03-12 12:11:23', 'JVM_NAME': 'TestiingAWS01', 'GCCOUNT': 10}, {'COUNT': 10, 'NAME': 'SRV0145GH', 'LASTOCCURED': '2018-03-12 12:11:42', 'JVM_NAME': 'TestiingAWS01', 'GCCOUNT': 10}, {'COUNT': 25, 'NAME': 'CLFRW0134W', 'LASTOCCURED': '2018-03-13 12:14:23', 'JVM_NAME': 'QAAWS02', 'GCCOUNT': 10}, {'COUNT': 10, 'NAME': 'SRV0145GH', 'LASTOCCURED': '2018-03-18 12:55:23', 'JVM_NAME': 'QAAWS02', 'GCCOUNT': 10}]

@AyodhyaAnkit Thanks, Can it be in pythonic one-liner!! – Rahul Trivedi Mar 21 '18 at 22:01 — Rahul Trivedi, Mar 21 '18 at 22:01

Python merge nested list of dictionary to single list of dictionary

4 Answers4