1

I want to merge two nested dicts with each other, sometimes they have the same amount of dicts in dicts sometimes not.

By merging I mean I want to keep everything in my already existing .json file and only update the new changed values from my example dict dict_01. If a key or whole dictionary does not exist, I want to add a new dict in dict if let's say 'name_03' does not yet exist.

import json
from pprint import pprint


json_filepath = 'database.json'

dict_01 = {"database": {"name_01": {"name": "name_01",
                                    "count": 10, 
                                    "size": "3"},
                        "name_03": {"name": "name_01",
                                    "count": 10, 
                                    "size": "3"}}}



with open(json_filepath, 'r', encoding="utf-8") as f:
    data = json.load(f)


pprint(data)

This seems to only replace my current dict. Therefore all existing information is deleted. Not what I'm after.

data.update(dict_01)  # That doesn't do what I want it to do


with open(json_filepath, 'w+') as f:
    json.dump(data)

Sample content of database.json file

{"database": {"name_01": {"count": 1,
                          "file_count": 1,
                          "folder_count": 0,
                          "hdd_master_name": "name_01_suffix",
                          "last_scanned": "14/04/20 15:55",
                          "name": "name_01",
                          "server_path": "root.txt",
                          "size": "0.18"},
              "name_02": {"all_types_count": 4,
                          "file_count": 8,
                          "folder_count": 0,
                          "hdd_master_name": "name_02_suffix",
                          "last_scanned": "14/04/20 15:55",
                          "name": "name_02",
                          "server_path": "...",
                          "size": "50.34"}}}

The result I'm looking for:

{"database": {"name_01": {"count": 10,
                          "file_count": 1,
                          "folder_count": 0,
                          "hdd_master_name": "name_01_suffix",
                          "last_scanned": "14/04/20 15:55",
                          "name": "name_01",
                          "server_path": "root.txt",
                          "size": "3"},
              "name_02": {"all_types_count": 4,
                          "file_count": 8,
                          "folder_count": 0,
                          "hdd_master_name": "name_02_suffix",
                          "last_scanned": "14/04/20 15:55",
                          "name": "name_02",
                          "server_path": "...",
                          "size": "50.34"},
              "name_03": {"name": "name_01",
                          "count": 10, 
                          "size": "3"}}}
BenjaminK
  • 653
  • 1
  • 9
  • 26

5 Answers5

2

Try:

import collections
def dict_merge(dct, merge_dct):
    for k, v in merge_dct.items():
        if (k in dct and isinstance(dct[k], dict)
            and isinstance(merge_dct[k], collections.Mapping)):
            dict_merge(dct[k], merge_dct[k])
        else:
            dct[k] = merge_dct[k]
    return dct

Full code

import json
import collections

with open("database.json", 'r', encoding="utf-8") as f:
    data = json.load(f)

dict_01 = {"database": {"name_01": {"name": "name_01",
                                    "count": 10, 
                                    "size": "3"},
                        "name_03": {"name": "name_01",
                                    "count": 10, 
                                    "size": "3"}}}

def dict_merge(dct, merge_dct):
    for k, v in merge_dct.items():
        if (k in dct and isinstance(dct[k], dict)
            and isinstance(merge_dct[k], collections.Mapping)):
            dict_merge(dct[k], merge_dct[k])
        else:
            dct[k] = merge_dct[k]
    return dct

out = dict_merge(data, dict_01)
print(out)
# {
#     "database": {
#         "name_01": {
#             "count": 10,
#             "file_count": 1,
#             "folder_count": 0,
#             "hdd_master_name": "name_01_suffix",
#             "last_scanned": "14/04/20 15:55",
#             "name": "name_01",
#             "server_path": "root.txt",
#             "size": "3"
#         },
#         "name_02": {
#             "all_types_count": 4,
#             "file_count": 8,
#             "folder_count": 0,
#             "hdd_master_name": "name_02_suffix",
#             "last_scanned": "14/04/20 15:55",
#             "name": "name_02",
#             "server_path": "...",
#             "size": "50.34"
#         },
#         "name_03": {
#             "name": "name_01",
#             "count": 10,
#             "size": "3"
#         }
#     }
# }

Source: Recursive dictionary merge in Python

Alexandre B.
  • 5,387
  • 2
  • 17
  • 40
  • Thank you this seems to work well. I only get following warning. `DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working` `and isinstance(merge_dct[k], collections.Mapping)):` Do you know anything about that? – BenjaminK Apr 19 '20 at 23:18
  • 1
    You can try replacing the `import collections` by `import collections.abc as collections`. This [discussion](https://stackoverflow.com/questions/53978542/how-to-use-collections-abc-from-both-python-3-8-and-python-2-7) deals with this deprecation. – Alexandre B. Apr 20 '20 at 07:18
1
dict_01["database"] = {**dict_01["database"], **dict_02["database"]}

I tested this.

Full working code:

dict_01 = {"database": {"name_01": {"name": "name_01",
                                    "count": 10, 
                                    "size": "3"},
                        "name_03": {"name": "name_01",
                                    "count": 10, 
                                    "size": "3"}}}

dict_02 = {"database": {"name_01": {"count": 1,
                          "file_count": 1,
                          "folder_count": 0,
                          "hdd_master_name": "name_01_suffix",
                          "last_scanned": "14/04/20 15:55",
                          "name": "name_01",
                          "server_path": "root.txt",
                          "size": "0.18"},
              "name_02": {"all_types_count": 4,
                          "file_count": 8,
                          "folder_count": 0,
                          "hdd_master_name": "name_02_suffix",
                          "last_scanned": "14/04/20 15:55",
                          "name": "name_02",
                          "server_path": "...",
                          "size": "50.34"}}}

dict_01["database"] = {**dict_01["database"], **dict_02["database"]} 
print(dict_01)

out:

{"database": {"name_01": {"count": 10,
                          "file_count": 1,
                          "folder_count": 0,
                          "hdd_master_name": "name_01_suffix",
                          "last_scanned": "14/04/20 15:55",
                          "name": "name_01",
                          "server_path": "root.txt",
                          "size": "3"},
              "name_02": {"all_types_count": 4,
                          "file_count": 8,
                          "folder_count": 0,
                          "hdd_master_name": "name_02_suffix",
                          "last_scanned": "14/04/20 15:55",
                          "name": "name_02",
                          "server_path": "...",
                          "size": "50.34"},
              "name_03": {"name": "name_01",
                          "count": 10, 
                          "size": "3"}}}
David Smolinski
  • 514
  • 3
  • 13
  • How will that help to update `data`? You're only working with `dict_01` and don't work with `data` at all? – BenjaminK Apr 19 '20 at 23:21
  • 1
    Amit Davidson's answer worked for you, and it doesn't mention data. Label your variables. I think data might be the same as dict_02 in my answer. I'm adding my assumptions to my answer. – David Smolinski Apr 20 '20 at 01:21
1

For merging 2 dicts I would use this:

def mergedicts(dict1, dict2):
   for k in set(dict1.keys()).union(dict2.keys()):
       if k in dict1 and k in dict2:
          if isinstance(dict1[k], dict) and isinstance(dict2[k], dict):
             yield (k, dict(mergedicts(dict1[k], dict2[k])))
          else:
             yield (k, dict2[k])
       elif k in dict1:
           yield (k, dict1[k])
       else:
          yield (k, dict2[k])

and:

dict1 = {1:{"a":"A"},2:{"b":"B"}}
dict2 = {2:{"c":"C"},3:{"d":"D"}}
print dict(mergedicts(dict1,dict2))

Which prints:

{1: {'a': 'A'}, 2: {'c': 'C', 'b': 'B'}, 3: {'d': 'D'}}
Amit Davidson
  • 326
  • 2
  • 3
  • Thank you this seems to work without any errors! The only thing that seemed important is the order so I had to switch it to `dict(mergedicts(dict2,dict1)` with my data. I this that comes with in my case `dict1`is my `dict_01`. – BenjaminK Apr 19 '20 at 23:29
1

You need to update data['database'] not data. Code goes as follows;

import json
from pprint import pprint

json_filepath = 'database.json'

dict_01 = {"database": {"name_01": {"name": "name_01",
                                    "count": 10, 
                                    "size": "3"},
                        "name_03": {"name": "name_01",
                                    "count": 10, 
                                    "size": "3"}}}



with open(json_filepath, 'r', encoding="utf-8") as f:
    data = json.load(f)


# here is the change
data['database'].update(dict_01['database'])

pprint(data)

with open(json_filepath, 'w+') as f:
    json.dump(data)

Hope it helps. :) Enjoy :)

Biswajit
  • 59
  • 6
  • Thank you for your answer. This is still overwriting a lot of data. I want to only update or add new values. And with that method, a lot of keys and values are deleted. – BenjaminK Apr 19 '20 at 23:11
  • Its still overwritting data means. I think you are trying to update parameters inside keys of database itself. Am i correct? For example you are trying to update `data['database']['name_01']` or so. Is is correct? – Biswajit Apr 20 '20 at 02:02
  • I think you need that way. Let give another Method to recursively update dictionary. – Biswajit Apr 20 '20 at 02:11
  • I have added an new answer, I think That will solve your purpose. – Biswajit Apr 20 '20 at 02:22
  • Let me know if it helps :) – Biswajit Apr 20 '20 at 02:23
1

@BenjaminK

If you need to update data recursively; for example; if db is: 

db_01 = {"database": {"name_01": {"name": "name_01",
                                    "count": 10, 
                                    "size": "3"},
                        "name_03": {"name": "name_01",
                                    "count": 10, 
                                    "size": "3"}}}

and dictionary you want to update is;

dict_to_update = {"database": {"name_03": {"father":{"name":"john"}}}}

The you need to do like following;

For Python2.X;

import collections

def update(db, dict2update):
    for k, v in dict2update.iteritems():
        if isinstance(v, collections.Mapping):
            db[k] = update(db.get(k, {}), v)
        else:
            db[k] = v
    return d

For Python3.X;

import collections.abc

def update(db, dict2update):
    for k, v in dict2update.items():
        if isinstance(v, collections.abc.Mapping):
            db[k] = update(db.get(k, {}), v)
        else:
            db[k] = v
    return db

You can update your database db_01 like following;

update(db_01, dict_01)

After doing this your db_01 will be;

{'database': {'name_01': {'name': 'name_01', 'count': 10, 'size': '3'},
  'name_03': {'name': 'name_01',
   'count': 10,
   'size': '3',
   'father': {'name': 'john'}}}}

Hope this helps. If not let me know. Happy Coding :)

Biswajit
  • 59
  • 6