Get nested arrays out of a dictionary

Question

I'm trying to parse the results of a simulation to extract all of the results that are numpy arrays. When simulating over a simple material, we might get a single dictionary with array values:

{'material1':array, 'material2':array, ...}

In more complex material simulations, we end up with nested dictionaries like:

{'material1': {'shellmaterial':array, 'corematerial':array}}

The depth of the nesting is arbitrary, and what I want to do is create a plot where all of the available arrays are returned to the user, named by their nesting. So for example,the above structure would end up like:

{'material1.shellmaterial' : array, 'material1.corematerial' : array}

We'd then put these in a dropdown menu. for easy viewing in a plot. Does anyone have a good way to iterate through an arbitrarily nested dictionary and return only the array type values with the new keys as shown above?

Results have to be stored this way for json compatibility, so I can't really go back and refactor to avoid this.

Sigh, what is it about this question that warrants downvotes exactly? — Adam Hughes, Jan 24 '15 at 23:33

user3467349 · Accepted Answer · 2015-01-24T23:39:51.767

Here is a function I used with some decision trees for language processing, it's not what you want, but it's the same basic idea.

def nodeMapForDict(d):
    node_map = []
    node_path = [] 
    def nodeRecursiveMap(d, node_path): 
        for key, val in d.items():
            if type(val) is not dict: node_map.append(node_path + [key]) 
            if type(val) is dict: 
                nodeRecursiveMap(val, node_path + [key])
    nodeRecursiveMap(d, node_path)
    return node_map

And here is one that should fit your use case:

def flattenDict(d):
    node_map = {}
    node_path = [] 
    def nodeRecursiveMap(d, node_path): 
        for key, val in d.items():
            if type(val) is not dict: node_map['.'.join(node_path + [key])] = val 
            if type(val) is dict: 
                nodeRecursiveMap(val, node_path + [key])
    nodeRecursiveMap(d, node_path)
    return node_map

example:

d= {'d': [1, 2, 3, 4], 'e': {'b': {'c': 1}}, 'a': {'b': 'c'}}
In [49]: flattenDict(d)
Out[49]: {'d': [1, 2, 3, 4], 'e.b.c': 1, 'a.b': 'c'}

I'd probably change type(val) is not dict to account for the second condition that it must also be an array. There are other values in our results that are not arrays, maybe a string or something, that I'll effectively ignore. But other than this, I think this will work as is. Thanks. I'm away from primary pc so will come back and mark as accepted after I've had a chance to put it in. — Adam Hughes, Jan 24 '15 at 23:41

score 0 · Answer 2 · answered Jan 25 '15 at 01:04

For completeness, here's the accepted answer with a type argument that lets you filter out types of values to retain. In my case, this is only array values, but such behavior is quite useful for parsing results of json files or in particular our simulation program's data.

def flattenDict(d, *types):
    node_map = {}
    node_path = [] 
    def nodeRecursiveMap(d, node_path): 
        for key, val in d.items():
            if type(val) in types: 
                node_map['.'.join(node_path + [key])] = val 
            if type(val) is dict: 
                nodeRecursiveMap(val, node_path + [key])
    nodeRecursiveMap(d, node_path)
    return node_map

For example to retain only ints and strings:

d= {'d': [1, 2, 3, 4], 'e': {'b': {'c': 1}}, 'a': {'b': 'c'}}
In [1]: flattenDict(d, int, basestring)
Out[2]: {'e.b.c': 1, 'a.b': 'c'}

Get nested arrays out of a dictionary

2 Answers2

Linked