How to handle dictionary attributes at arbitrarily nested depths?

Question

I'm parsing a json -> dict object that has values nested at arbitrary depths. For example, the outer level are the fields (the data is from a customer service ticketing system) such as 'name', 'sprint_team', etc... But then for some fields they have a nested dict that will have values like 'id', 'value', etc...

I'm pulling it all down to place in a flat file to then copy over to AWS Redshift, but I'm trying to figure out an efficient way to handle all the different nested levels of values I might need. I was thinking of using a lookup table such as:

lookup = {
'customfield_12700': ('sprint_team', 'name'),
'customfield_13208': ('department', 'sub_department', 'name') 
}

where the value[0] is always the lookup name, and then there can be n number of extra params that will be my levels of nesting. The reason I'm using a lookup table in the first place is because Jira uses customfields, and I need to map them to the column names in our cloud database. For example, on the sprint_team field I need the name attribute for my flat file, etc...

How could I create a function that will take a dictionary object, and then use the lookup table to return the value I need, regardless of how deeply nested it is?

Are you looking for something like this? http://stackoverflow.com/questions/10756427/loop-through-all-nested-dictionary-values — idjaw, Feb 19 '16 at 21:28
Recursion was my first thought as well, but I usually shy away from using recursion in code that is going to be used in production. — flybonzai, Feb 19 '16 at 21:49
If you have a recursive structure that needs to be flattened then there is no reason to shy away from recursive parsing. It's not the best way to achieve your ends, it is **the right way** to meet those ends. Since your question *requires* moving hierarchical data to a standard RDBMs, you should put much thought into properly normalizing your input before stashing. — msw, Feb 19 '16 at 22:35

DNA · Accepted Answer · 2016-02-19T22:28:10.280

Something like this?

This uses your lookup table idea, and just navigates down through the dicts one level at a time.

>>> data = {"foo": {"bar": {"baz": 1}}}
>>> lookup = {
...   'customfield': ("foo", "bar", "baz" )
... }

>>> def nest(data, name):
...   path = lookup[name]
...   d = data
...   for p in path:
...     d = d[p]
...   return d
...
>>> nest(data, 'customfield')
1

This can also be done recursively, which should be fine unless your dicts are very deeply nested!

def nest(data, path):
      if len(path) == 0:
          return data
      else:
          return nest(data[path[0]], path[1:])

def find(data, name):
    return nest(data, lookup[name])

>>>find(data, 'customfield')
1

Alternatively, using a funky functional approach:

>>> from operator import itemgetter
>>> from functools import reduce
>>> def compose(*functions):
...     return reduce(lambda f, g: lambda x: f(g(x)), functions, lambda x: x)

>>> nest = compose(*[itemgetter(name) for name in reversed(lookup['customfield'])])
>>> nest(data)
1

itemgetter provides a function that accesses an attribute with the specified name. compose composes these functions together to make a single function that extracts the nested data.

You could also pre-generate these functions and store them in lookup instead of the paths, so they can be used directly when required.

(We have to define our own compose function since Python doesn't have one built-in).

How to handle dictionary attributes at arbitrarily nested depths?

1 Answers1