Merging Python list of dicts grouped by key value

Question

I want to merge two dicts on the id. Dict x contains many distinct ids and different row counts per each id. Dict y contains multiple key values, and always has less rows than dict x.

x = [{'costgroup': '1', 'POC1': '2', 'post': '5','id': '1'},
    {'costgroup': '2', 'POC1': '1', 'post': '4','id': '1'},
    {'costgroup': '3', 'POC1': '5', 'post': '2', 'id': '1'},
    {'costgroup': '1', 'POC1': '2', 'post': '5','id': '2'},
    {'costgroup': '2', 'POC1': '1', 'post': '4','id': '2'},
    {'costgroup': '3', 'POC1': '5', 'post': '2', 'id': '2'},
    {'costgroup': '3', 'POC1': '5', 'post': '2', 'id': '2'}] 


 y = [{'id': '1', 'laminate': 'D'}, 
     { 'id':'2', 'laminate': T'}]

The output that I want is the following:

z = [{'costgroup': '1', 'POC1': '2', 'post': '5','id': '1','laminate':'D'},
    {'costgroup': '2', 'POC1': '1', 'post': '4','id': '1','laminate': 'D'},
   {'costgroup': '3', 'POC1': '5', 'post': '2', 'id': '1','laminate': 'D'},
   {'costgroup': '1', 'POC1': '2', 'post': '5','id': '2','laminate': 'T'},
    {'costgroup': '2', 'POC1': '1', 'post': '4','id': '2','laminate': 'T'},
   {'costgroup': '3', 'POC1': '5', 'post': '2', 'id': '2','laminate': 'T'},
   {'costgroup': '3', 'POC1': '5', 'post': '2', 'id': '2','laminate': 'T'}]

This is easy to achieve using pandas

dfx  = pd.DataFrame(x)
dfy  = pd.DataFrame(y)
pd.merge(dfx,dfy, how ='left', left_on = 'id', right_on = 'id' )

But, I am going to apply this using an AWS Lambda function and I don't want to have the overhead of pandas and the output needs to be a dict. I tried the code below which gets me closer, but then i had to add something to find the distinct values of the ID and iterate through them.But,still don't have the output that I need.

valuelist = ['1']
def copyf(dictlist, key, valuelist):
      return [d for d in dictlist if d[key] in valuelist]

y1 = copyf(y, 'id', valuelist)
x1 = copyf(x, 'id', valuelist)
y1.append(x1)

The above provides this output, which is interesting but not what I need.

[{'distance': '2', 'id': '1', 'laminate': 'D'},
 [{'POC1': '2', 'costgroup': '1', 'id': '1', 'post': '5'},
  {'POC1': '1', 'costgroup': '2', 'id': '1', 'post': '4'},
  {'POC1': '5', 'costgroup': '3', 'id': '1', 'post': '2'}]]

Are the `ids` unique in `y`? – user2390182 Mar 18 '17 at 17:12 — user2390182, Mar 18 '17 at 17:12

ewcz · Answer 1 · 2017-03-18T17:20:18.637

import copy

x = [{'costgroup': '1', 'POC1': '2', 'post': '5','id': '1'},
    {'costgroup': '2', 'POC1': '1', 'post': '4','id': '1'},
    {'costgroup': '3', 'POC1': '5', 'post': '2', 'id': '1'},
    {'costgroup': '1', 'POC1': '2', 'post': '5','id': '2'},
    {'costgroup': '2', 'POC1': '1', 'post': '4','id': '2'},
    {'costgroup': '3', 'POC1': '5', 'post': '2', 'id': '2'},
    {'costgroup': '3', 'POC1': '5', 'post': '2', 'id': '2'}]


y = [{'id': '1', 'laminate': 'D'},
     { 'id':'2', 'laminate': 'T'}
]

#create the id->laminate mapping
m = { d['id']: d['laminate'] for d in y }

#create the final output
z = []
for d in x:
    #make a copy of the dictionary from x so that the input data
    #is not overwritten
    item = dict(d) #copy.deepcopy(d)
    item.update({'laminate': m[d['id']]})
    z.append(item)

print(z)

this produces

[
 {'laminate': 'D', 'post': '5', 'POC1': '2', 'id': '1', 'costgroup': '1'},
 {'laminate': 'D', 'post': '4', 'POC1': '1', 'id': '1', 'costgroup': '2'},
 {'laminate': 'D', 'post': '2', 'POC1': '5', 'id': '1', 'costgroup': '3'},
 {'laminate': 'T', 'post': '5', 'POC1': '2', 'id': '2', 'costgroup': '1'},
 {'laminate': 'T', 'post': '4', 'POC1': '1', 'id': '2', 'costgroup': '2'},
 {'laminate': 'T', 'post': '2', 'POC1': '5', 'id': '2', 'costgroup': '3'},
 {'laminate': 'T', 'post': '2', 'POC1': '5', 'id': '2', 'costgroup': '3'}]

score 2 · Accepted Answer · edited May 23 '17 at 12:09

def merge(d1, d2):
    """Given two dicts, merge them into a new dict as a shallow copy."""
    result = d1.copy()
    result.update(d2)
    return result

result = [merge(d1, d2) for d1 in x for d2 in y if d1["id"] == d2["id"]]
print(result)

Gives

[{'POC1': '2', 'costgroup': '1', 'id': '1', 'laminate': 'D', 'post': '5'},
 {'POC1': '1', 'costgroup': '2', 'id': '1', 'laminate': 'D', 'post': '4'},
 {'POC1': '5', 'costgroup': '3', 'id': '1', 'laminate': 'D', 'post': '2'},
 {'POC1': '2', 'costgroup': '1', 'id': '2', 'laminate': 'T', 'post': '5'},
 {'POC1': '1', 'costgroup': '2', 'id': '2', 'laminate': 'T', 'post': '4'},
 {'POC1': '5', 'costgroup': '3', 'id': '2', 'laminate': 'T', 'post': '2'},
 {'POC1': '5', 'costgroup': '3', 'id': '2', 'laminate': 'T', 'post': '2'}]

Merge function from here: How to merge two Python dictionaries in a single expression?

There's a more concise syntax in Python 3.5 but you're on 2.7.

score 1 · Answer 3 · answered Mar 18 '17 at 17:36

1

z = []
for dx in x:
    for dy in y:
        if dx['id'] == dy['id']:
            z.append(dict(dx.items () + dy.items()))

print z

answered Mar 18 '17 at 17:36

plasmon360

4,109
1
16
19

Merging Python list of dicts grouped by key value

3 Answers3