How to iterate through an N-level nested dictionary in Python?

Question

I find myself making multilevel dictionaries quite a bit. I always have to write very verbose code to iterate through all the levels of the dictionaries with a lot of temporary variables.

Is there a way to generalize this function to iterate through multiple levels instead of hardcoding in and manually specifying how many levels there are?

def iterate_multilevel_dictionary(d, number_of_levels):
    # How to auto-detect number of levels? 
    # number_of_levels = 0
    if number_of_levels == 1:
        for k1, v1 in d.items():
            yield k1, v1
    if number_of_levels == 2:
        for k1, v1 in d.items():
            for k2, v2 in v1.items():
                yield k1, k2, v2
    if number_of_levels == 3:
        for k1, v1 in d.items():
            for k2, v2 in v1.items():
                for k3, v3 in v2.items():
                    yield k1, k2, k3, v3
                    
# Level 1
d_level1 = {"a":1,"b":2,"c":3}
for items in iterate_multilevel_dictionary(d_level1, number_of_levels=1):
    print(items)
# ('a', 1)
# ('b', 2)
# ('c', 3)

# Level 2
d_level2 = {"group_1":{"a":1}, "group_2":{"b":2,"c":3}}
for items in iterate_multilevel_dictionary(d_level2, number_of_levels=2):
    print(items)
#('group_1', 'a', 1)
#('group_2', 'b', 2)
#('group_2', 'c', 3)

# Level 3
d_level3 = {"collection_1":d_level2}
for items in iterate_multilevel_dictionary(d_level3, number_of_levels=3):
    print(items)
# ('collection_1', 'group_1', 'a', 1)
# ('collection_1', 'group_2', 'b', 2)
# ('collection_1', 'group_2', 'c', 3)

This is one of the rare strong-use cases for recursion, since nesting is rarely deep enough to cause stack issues. Have you tried a recursive approach? — Carcigenicate, Jul 22 '21 at 19:34
Not off-hand. A typical recipe is to pass the "current level" to the recursive function, then check if that layer is in fact a layer, and if it is, pass each sub-piece to a recursive call, else, operate on that data. That may not translate *directly* to your exact case here, but the answer to iterating recursive structures (which I think nested dictionaries would be considered) is often to use recursion, so that's what I thought of here. — Carcigenicate, Jul 22 '21 at 19:39
Like [this](https://stackoverflow.com/a/10756547/3000206). Again, not directly related, but that's an often-used pattern that may find relevance in your problem. — Carcigenicate, Jul 22 '21 at 19:41

S.B · Answer 1 · 2021-07-27T09:03:31.250

I've written this after I saw @VoNWooDSoN's answer. I turned it into an iterator instead of printing inside the function and a little bit of changes to make it more readable. So see his original answer here.

def flatten(d, base=()):
    for k, v in d.items():
        if isinstance(v, dict):
            yield from flatten(v, base + (k,))
        else:
            yield base + (k, v)

1- yielding instead of printing.

2- isinstance() instead of type so that subclasses of dict can also work. You could also use MutableMapping from typing module instead of dict to make it more generic.

3- IMO , getting (k, v) pairs from .items() is much more readable than k and d[k].

More generic ?

Do you wanna expand this to even more generic which CAN(not have to, like the solution in the OP) accept the number of depths just in case?

Consider these examples:

d_level1 = {"a": 1, "b": 2, "c": 3}
d_level2 = {"group_1": {"a": 1}, "group_2": {"b": 2, "c": 3}}
d_level3 = {"collection_1": d_level2}

for items in flatten(d_level3):
    print(items)
print('------------------------------')
for items in flatten(d_level3, depth=0):
    print(items)
print('------------------------------')
for items in flatten(d_level3, depth=1):
    print(items)
print('------------------------------')
for items in flatten(d_level3, depth=2):
    print(items)

output:

('collection_1', 'group_1', 'a', 1)
('collection_1', 'group_2', 'b', 2)
('collection_1', 'group_2', 'c', 3)
------------------------------
('collection_1', {'group_1': {'a': 1}, 'group_2': {'b': 2, 'c': 3}})
------------------------------
('collection_1', 'group_1', {'a': 1})
('collection_1', 'group_2', {'b': 2, 'c': 3})
------------------------------
('collection_1', 'group_1', 'a', 1)
('collection_1', 'group_2', 'b', 2)
('collection_1', 'group_2', 'c', 3)

depth=None doesn't consider the depth (still works like you want at the first place). But now by specifying depths from 0 to 2 you can see that we are able to iterate how deep we want. here is the code:

def flatten(d, base=(), depth=None):
    for k, v in d.items():
        if not isinstance(v, dict):
            yield base + (k, v)
        else:
            if depth is None:
                yield from flatten(v, base + (k,))
            else:
                if depth == 0:
                    yield base + (k, v)
                else:
                    yield from flatten(v, base + (k,), depth - 1)

VoNWooDSoN · Answer 2 · 2021-07-26T20:30:03.907

Here's a quick and dirty solution for you:

d_level1 = {"a":1,"b":2,"c":3}
d_level2 = {"group_1":{"a":1}, "group_2":{"b":2,"c":3}}
d_level3 = {"collection_1":d_level2}

def flatten(d_in, base=()):
    for k in d_in:
        if type(d_in[k]) == dict:
            flatten(d_in[k], base+(k,))
        else:
            print(base + (k, d_in[k]))

flatten(d_level1)
# ('a', 1)
# ('b', 2)
# ('c', 3)

flatten(d_level2)
#('group_1', 'a', 1)
#('group_2', 'b', 2)
#('group_2', 'c', 3)

flatten(d_level3)
# ('collection_1', 'group_1', 'a', 1)
# ('collection_1', 'group_2', 'b', 2)
# ('collection_1', 'group_2', 'c', 3)

Be aware!! Python has a recursion limit of about 1000! So, when using recursion in python think very carefully what you're trying to do and be prepared to catch a RuntimeError if you call a recursive function like this.

EDIT: With comments I realized that I'd made a mistake where I did not add the key to the level1 dict output and that I was using a mutable structure as a default argument. I added these and parens in the print statement and reposted. The output now matches the OP's desired output and uses better and modern python.

it's better not to set mutable values as the parameter default values, use an empty tuple for the d's default value. — Kasra, Jul 26 '21 at 18:26
It doesn't report the keys of the level 1 dictionary because it's a bug. I'll fix it. Also, using a mutable collection (a list) as a default parameter is a bad idea. I'll also fix that. Standby for an edit. — VoNWooDSoN, Jul 26 '21 at 20:19

score 2 · Accepted Answer · answered Jul 26 '21 at 18:11

try out this code

it also supports a combination of levels

from typing import List, Tuple


def iterate_multilevel_dictionary(d: dict):
    dicts_to_iterate: List[Tuple[dict, list]] = [(d, [])]
    '''
    the first item is the dict object and the second object is the prefix keys 
    '''
    while dicts_to_iterate:
        current_dict, suffix = dicts_to_iterate.pop()
        for k, v in current_dict.items():
            if isinstance(v, dict):
                dicts_to_iterate.append((v, suffix + [k]))
            else:
                yield suffix + [k] + [v]


if __name__ == '__main__':
    d_level1 = {"a": 1, "b": 2, "c": 3}
    print(f"test for {d_level1}")
    for items in iterate_multilevel_dictionary(d_level1):
        print(items)
    d_level2 = {"group_1": {"a": 1}, "group_2": {"b": 2, "c": 3}}
    print(f"test for {d_level2}")
    for items in iterate_multilevel_dictionary(d_level2):
        print(items)

    d_level3 = {"collection_1": d_level2}
    print(f"test for {d_level3}")
    for items in iterate_multilevel_dictionary(d_level3):
        print(items)

    d_level123 = {}
    [d_level123.update(i) for i in [d_level1, d_level2, d_level3]]
    print(f"test for {d_level123}")
    for items in iterate_multilevel_dictionary(d_level123):
        print(items)

the outputs is:

test for {'a': 1, 'b': 2, 'c': 3}
['a', 1]
['b', 2]
['c', 3]
test for {'group_1': {'a': 1}, 'group_2': {'b': 2, 'c': 3}}
['group_2', 'b', 2]
['group_2', 'c', 3]
['group_1', 'a', 1]
test for {'collection_1': {'group_1': {'a': 1}, 'group_2': {'b': 2, 'c': 3}}}
['collection_1', 'group_2', 'b', 2]
['collection_1', 'group_2', 'c', 3]
['collection_1', 'group_1', 'a', 1]
test for {'a': 1, 'b': 2, 'c': 3, 'group_1': {'a': 1}, 'group_2': {'b': 2, 'c': 3}, 'collection_1': {'group_1': {'a': 1}, 'group_2': {'b': 2, 'c': 3}}}
['a', 1]
['b', 2]
['c', 3]
['collection_1', 'group_2', 'b', 2]
['collection_1', 'group_2', 'c', 3]
['collection_1', 'group_1', 'a', 1]
['group_2', 'b', 2]
['group_2', 'c', 3]
['group_1', 'a', 1]

using recursion is another approach but I thought writing without recursion is more challenging and more efficient :)

score 0 · Answer 4 · answered Aug 01 '21 at 00:17

A linear Solution for identifying depth of a dictionary.

d={'a':{1:1,2:2},"b":0,'c':"{}"}
print(d)
s=str(d)

dictionary_stack,dictionary_depth=0,0
def push():
    global dictionary_depth
    global dictionary_stack
    dictionary_stack+=1
    dictionary_depth=max(dictionary_depth,dictionary_stack)

def pop():
    global dictionary_stack
    dictionary_stack-=1

string_safety=False
for c in s:
    if c =="'":
        string_safety=not(string_safety)
    
    if not(string_safety) and c =='{':
        push()
    
    if not(string_safety) and c =='}':
        pop()


print(dictionary_depth)

Output:

{'a': {1: 1, 2: 2}, 'b': 0, 'c': '{}'}

2

score 0 · Answer 5 · answered Aug 25 '23 at 07:04

nested_dict = {'A': {'key_B': 'value_B'},
           'B': {'key_C': 'value_C'},
           'C': {'key_C': {'key_D':'value_D'}}
           }


def loop_nested(dictionary: dict):
  for key in dictionary:
  value = dictionary[key]
  print(key)
  if isinstance(value, dict):
    loop_nested(value)
  else:
   print(value)


loop_nested(nested_dict)

How to iterate through an N-level nested dictionary in Python?

5 Answers5

More generic ?