Merge dictionaries in list based on top level keys

Question

I have multiple dictionaries in a list, and the structure of the dictionaries are like this:

dict1 = {'first': [{'att1': 'abc',
                    'att2': '123',
                    'att3': 'abc123'},
                   {'att1': 'abc',
                    'att2': '123',
                    'att3': 'abc123'}],
         'second': [{'att1': 'abc',
                     'att2': '123'},
                    {'att1': 'abc',
                     'att2': '123'}],
         'third': [{'att4': 'bca'}]}

dict2 = {'first': [{'att1': 'abc',
                    'att2': '123',
                    'att3': 'abc123'},
                   {'att1': 'abc',
                    'att2': '123',
                    'att3': 'abc123'}],
         'second': [{'att1': 'abc',
                     'att2': '123'},
                    {'att1': 'abc',
                     'att2': '123'}]}
dict_list =  [dict1, dict2]

I want to merge the dictionaries in the list based on the highest level keys 'first', 'second' and 'third' if they exist in the dictionaries, so my desired output would look like this:

merged = {'first': [{'att1': 'abc',
                      'att2': '123',
                      'att3': 'abc123'},
                     {'att1': 'abc',
                      'att2': '123',
                      'att3': 'abc123'},
                     {'att1': 'abc',
                      'att2': '123',
                      'att3': 'abc123'},
                     {'att1': 'abc',
                      'att2': '123',
                      'att3': 'abc123'}],
         'second': [{'att1': 'abc',
                     'att2': '123'},
                    {'att1': 'abc',
                     'att2': '123'},
                    {'att1': 'abc',
                     'att2': '123'},
                    {'att1': 'abc',
                     'att2': '123'}],
          'third': [{'att4': 'bca'}]}

The dictionaries are created from dataframes using the pandas function to_dict. If merging json objects based on keys is easier or just as easy, this is also an option as i could use to_json instead. Is there any smart pythonic way of doing this? Any help is appreciated.

If this comes from dataframes, it is almost certainly better/easier to work with DFs instead. JSON (from to_json) is a string. You will need to load them into dicts anyway to process. But in any case, you should show what have you tried so far (i.e. your code). — buran, Oct 14 '22 at 08:47
https://stackoverflow.com/questions/38987/how-do-i-merge-two-dictionaries-in-a-single-expression — Markus Schmidgall, Oct 14 '22 at 08:49
By the way, how are they created from DFs, given that arrays are not same length? It throws error if you try to create DF from them — buran, Oct 14 '22 at 08:52
I have tried that solution, using dict1 | dict2, but this results in an overwrite of the values in att1, att2 etc. I want to keep everything in both dictionaries and only merge on the top level keys. — pyth0nEiken, Oct 14 '22 at 08:56
They are created from dataframes, where 'first' is one dataframe, 'second' is one dataframe, and 'third' is one dataframe. I am using df.to_dict(orient='records'), so that each of the dictionaries in the arrays are one row from the dataframe with its corresponding column label — pyth0nEiken, Oct 14 '22 at 08:59

Rahul K P · Answer 1 · 2022-10-14T09:32:59.493

0

You can do this,

merged = {}
for k, v in dict1.items():
    if dict2.get(k):
        # To avoid the changing the original value kept `v[:]`
        merged.setdefault(k, v[:]).append(dict2.get(k))
    else:
        merged[k] = v

Output:

{'first': [{'att1': 'abc', 'att2': '123', 'att3': 'abc123'},
  {'att1': 'abc', 'att2': '123', 'att3': 'abc123'},
  [{'att1': 'abc', 'att2': '123', 'att3': 'abc123'},
   {'att1': 'abc', 'att2': '123', 'att3': 'abc123'}]],
 'second': [{'att1': 'abc', 'att2': '123'},
  {'att1': 'abc', 'att2': '123'},
  [{'att1': 'abc', 'att2': '123'}, {'att1': 'abc', 'att2': '123'}]],
 'third': [{'att4': 'bca'}]}

edited Oct 14 '22 at 09:32

answered Oct 14 '22 at 09:21

Rahul K P

15,740
4
35
52

Yes, this solution would work fine if my dictionaries was accessible as separate variables. However, keep in mind that my dictionaries are in an array, and not accessible as variables dict1 and dict2. I only have dict_list variable with dictionaries inside to work with – pyth0nEiken Oct 14 '22 at 09:54
@pyth0nEiken the approach I made according to your question. It will be great if you update the question accordingly. And if you have a list of dictionaries, so you need to iterate through the list and assign accordingly. – Rahul K P Oct 14 '22 at 12:03

Merge dictionaries in list based on top level keys

1 Answers1