In Python, how to generate nested dict from dotted json file?

Question

I have a json file like:

{
    "a": 0.7615894039735099,
    "a.b": 0.7152317880794702,
    "a.c": 0.026490066225165563,
    "a.b.d": 0.0001,
    "f": 0.002,
    "f.g": 0.00003,
    "h.p.q": 0.0004
}

say the whole dict is called "root"

I want to use it this way

if "c" in root["a"] and if root["a"]["c"] > 0.0002:
   print("something")

Can anyone help with this? Thank you so much!

The data structure of regular python dictionary can't support your use case. Since you're storing a `float` in `root["a"]`, it will return an TypeError when you execute the statement `"c" in root["a"]` as `float` type is not iterable. — Orix Au Yeung, Mar 04 '20 at 06:18
I think, first you need to re-structure the JSON data to make it nested. — Harun Yilmaz, Mar 04 '20 at 06:24
If you could by any means store the keys say 'a' and 'c' in an array You can use this ```root['.'.join(['a','c'])]``` — Biswa Soren, Mar 04 '20 at 06:33
[Dot notation to Json in python](https://stackoverflow.com/q/25389875/674039) — wim, Jun 22 '22 at 02:14

A. Rom · Accepted Answer · 2020-03-04T10:29:17.953

As keys are hashed- there is no semantic meaning to the existence or lack of a dot from a string key, and an attempt to access root["a"]["c"] would result in a TypeError exception as mentioned before. You could, however, re-structure the dictionary to have the nested structure you're looking for. The code would look roughly like that:

root = {
    "a": 0.7615894039735099,
    "a.b": 0.7152317880794702,
    "a.c": 0.026490066225165563,
    "a.b.d": 0.0001,
    "f": 0.002,
    "f.g": 0.00003,
    "h.p.q": 0.0004
}

result = {}
for key, value in root.items():
     if not isinstance(key, str):
         result[key] = value
     is_nested = "." in key
     nesting_clash = any([k.startswith(key) for k in root if k != key])
     if nesting_clash:
         print(f"key {key} has nesting clash ; replacing with {key}._v")
         # continue # fixed after comment
         key = f"{key}._v"
         is_nested = True
     if not is_nested:
         result[key] = value
     key_parts = key.split(".")
     tmp = result
     for idx, key_part in enumerate(key_parts):
         default_value = {} if idx < len(key_parts) - 1 else value
         tmp[key_part] = tmp.get(key_part, default_value)
         tmp = tmp[key_part]

Note: you'd have to either discard keys that clash (e.g. "a" and "a.b") or to create a default behavior for them. In my example I decided to discard them.

EDIT: I've replaced the skip with a key replacement of ._v . this way you can keep all values and reconstruct the original dictionary/JSON by using the following function:

def unnest_dict(d, sep="."):
    result = {}
    for key, val in d.items():
        if not isinstance(val, dict):
            result[key] = val
            continue
        if "_v" in val:
            result[key] = val.pop("_v")
        unnested_val = unnest_dict(val, sep)
        for k, v in unnested_val.items():
            result[sep.join([key, k])] = v
    return result

Thanks A.Rom. But to me, I must have root["a"] to return the property value. So I guess I can only use an class object to create a tree in this situation? — shenmufeng, Mar 04 '20 at 08:20
Could you please be more specific about how to re-construct the original nested dict? If it works, I could try this way.@A. Rom — shenmufeng, Mar 04 '20 at 08:26
@shenmufeng - Edited and allowed keeping the original values + function to reconstruct the original dict. — A. Rom, Mar 04 '20 at 10:30
Great solution! Thank you Rom. I will take your suggestion and thank you so much for your help. — shenmufeng, Mar 05 '20 at 02:21

score 1 · Answer 2 · answered Mar 04 '20 at 06:56

Look at the below snippet.


def merge(a, b, path=None):
    '''merges b into a'''
    if path is None: path = []
    for key in b:
        if key in a:
            if isinstance(a[key], dict) and isinstance(b[key], dict):
                merge(a[key], b[key], path + [str(key)])
            elif a[key] == b[key]:
                pass # same leaf value
            else:
                raise Exception('Conflict at %s' % '.'.join(path + [str(key)]))
        else:
            a[key] = b[key]
    return a


def convert_dotted_json(dottedJson):
    '''
    parameter
        dottedJson : dict type
    '''
    root = {}

    for key in dottedJson:
        split_key = key.split(".");
        split_key.reverse()
        value = dottedJson [key]
        curr = {split_key[0]: value};
        for ind in range(1, len(split_key)):
            curr = {split_key[ind]:curr}
        root = merge(root, curr)

    return root


test = {
    "a.b.c" : 0.026490066225165563,
    "a.b.d" : 0.0001,
    "f.g": 0.00003,
    "f.h": 0.00003,
    "h.p.q": 0.0004
}

# print(convert_dotted_json(test))

Copied merge function from Andrew Cook's answer.

Thanks Vivek. But I do need to use root["a"]'s value. It seems we could not achieve this goal?@Vivek Kundariya — shenmufeng, Mar 04 '20 at 08:24

In Python, how to generate nested dict from dotted json file?

2 Answers2