Using map and lambda to handle nested dicts

Question

I currently have a dict like this (assume many countries, states, and cities):

'USA': {
    'Texas': {
        'Austin': {
            '2017-01-01': 169,
            '2017-02-01': 231
        },
        'Houston': {
            '2017-01-01': 265,
            '2017-02-01': 310
        }
    }

I want to create a new dict "grouping by" only country and date, filtering for a given state, so the result would be:

'USA': {
            '2017-01-01': 434,
            '2017-02-01': 541

    }

I can do this by looping over each layer of the dict, but it's hard to read. Is there a way to do this with lambda/map functions instead?

Also, we are unable to use pandas dataframes for other reasons, so I can't use that groupby feature.

This is the first time I've ever seen someone who wanted to use `map` and lambdas to make something _more_ readable. I don't think what you're asking is possible... — Aran-Fey, Mar 12 '18 at 17:39

jpp · Answer 1 · 2018-03-12T18:34:56.773

If you only want to extract the lowest level values of your nested dictionary, this can be achieved using a generator.

The below generator is a slightly modified version of one written by @Richard.

You can then combine this with collections.defaultdict to obtain your desired result.

from collections import defaultdict

def NestedDictValues(d):
    for k, v in d.items():
        if isinstance(v, dict):
            yield from NestedDictValues(v)
        else:
            yield (k, v)

def sumvals(lst):
    c = defaultdict(int)
    for i, j in lst:
        c[i] += j
    return dict(c)

d = {'USA': sumvals(NestedDictValues(s))}

# {'USA': {'2017-01-01': 434, '2017-02-01': 541}}

score 0 · Answer 2 · answered Mar 12 '18 at 17:46

I believe that in this case, it is much cleaner to use recursion than a map or reduce function:

import re
import itertools
s = {'USA': {
'Texas': {
    'Austin': {
        '2017-01-01': 169,
        '2017-02-01': 231
    },
    'Houston': {
        '2017-01-01': 265,
        '2017-02-01': 310
    }
   }
 }
}
def get_dates(d):
  val = [(a, b) if isinstance(b, int) and re.findall('\d+-\d+-\d+', a) else get_dates(b) for a, b in d.items()]
  return [i for b in val for i in b] if not all(isinstance(i, tuple) for i in val) else val

last_data = {a:{c:sum(g for _, g in h) for c, h in itertools.groupby(sorted(get_dates(b), key=lambda x:x[0]), key=lambda x:x[0])} for a, b in s.items()}

Output:

{'USA': {'2017-02-01': 541, '2017-01-01': 434}}

score 0 · Answer 3 · answered Mar 12 '18 at 18:41

Here's a way to possibly simplify your existing code using collections.Counter. Suppose your source dictionary is named d:

from collections import Counter
my_state='Texas'
mapped = {
    country: [Counter(d[country][my_state][city]) for city in d[country][my_state]]
    for country in d
}
print(mapped)
#{'USA': [Counter({'2017-01-01': 265, '2017-02-01': 310}),
#  Counter({'2017-01-01': 169, '2017-02-01': 231})]}

This mapped your original dictionary in to one of the form {country: list_of_counters}.

Now you can use operator.add() to reduce this list:

from operator import add
for country in mapped:
    print("{country}: {sums}".format(country=country, sums=reduce(add, mapped[country])))
#USA: Counter({'2017-02-01': 541, '2017-01-01': 434})

Or as map/reduce:

map(lambda country: {country: reduce(add, mapped[country])}, mapped)
[{'USA': Counter({'2017-01-01': 434, '2017-02-01': 541})}]

If you prefer to have dicts instead of Counters:

map(lambda country: {country: dict(reduce(add, mapped[country]))}, mapped)
#[{'USA': {'2017-01-01': 434, '2017-02-01': 541}}]

Using map and lambda to handle nested dicts

3 Answers3