List Comprehension to Flatten a Dictionary of Dictionaries

Question

I have a dictionary of dictionaries:

my_dict = {
    'a': {(1,2): True,
          (1,3): False},
    'b': {(1,4): True,
          (2,3): False}
}

The dictionary is always of this form, but every 'child' dictionary has a different set of keys: my_dict['a'][(1,2)] exists, but that doesn't mean my_dict['b'][(1,2)] also exists.

I want a list (in no particular order) of the boolean values:

[True, False, True, False]

I am trying to use a single list comprehension to accomplish this:

[my_dict[letter][pair] for pair in my_dict[letter] for letter in my_dict]

This raises an error:

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-142-dc1565efcdc8> in <module>()
      6 }
      7 
----> 8 [my_dict[letter][pair] for pair in my_dict[letter] for letter in my_dict]
KeyError: (2, 3)

It appears to be looking for (2,3) in both my_dict['a'] and my_dict['b']. I thought the comprehension I wrote would only look for the keys in the appropriate dictionary.

I've seen this solution which can work to flatten any nested dictionary. I also know I could brute force it with imperative loops. I am just trying to understand why the list comprehension isn't working the way I have it written.

score 5 · Accepted Answer · answered Jan 18 '15 at 22:20

5

You want to loop over the values of the values:

[v for nested in outer.itervalues() for v in nested.itervalues()]

Note that the loops need to be ordered the way you'd nest them; outer loop first:

for nested in outer.itervalues():
    for v in nested.itervalues():
        # use v

You had the order mixed up; your code only gave KeyError because you had a pre-existing letter global.

Demo:

>>> my_dict = {
...     'a': {(1,2): True,
...           (1,3): False},
...     'b': {(1,4): True,
...           (2,3): False}
... }
>>> [v for nested in my_dict.itervalues() for v in nested.itervalues()]
[True, False, False, True]

answered Jan 18 '15 at 22:20

Martijn Pieters

1,048,767
296
4,058
3,343

Might want to note that the OP may not get `[True, False, True, False]` as they seem to expect - oh - never mind - didn't see the *in no particular order* - in which case might as well just use a `Counter` instead of a list :) – Jon Clements Jan 18 '15 at 22:24
Thanks for the explanation, I appreciate it. It is hard to wrap my head around the order of nesting when you are doing list comprehensions, though it makes sense when you think if it like any other nested for loop. for some reason I think it makes more sense going the other way. – elsherbini Jan 18 '15 at 22:37
Also, this seemed to work just fine: `[my_dict[letter][pair] for letter in my_dict for pair in my_dict[letter]]` (just switching the order) – elsherbini Jan 18 '15 at 22:43
@elsherbini that works too, just not as efficiently. :-) But yes, getting the order right is key. – Martijn Pieters Jan 18 '15 at 22:51

Linguist · Answer 2 · 2015-01-18T23:28:08.990

As elsherbini said,

[my_dict[letter][pair] for letter in my_dict for pair in my_dict[letter]]

This also works:

[little_dict[k] for little_dict in [my_dict[letter] for letter in my_dict] for k in little_dict]

Both produce [True, False, False, True].

You want to understand why your original try doesn't work.

[my_dict[letter][pair] for pair in my_dict[letter] for letter in my_dict]

The only reason this runs at all is that you must have had letter previously defined, perhaps a definition left over from previously running some similar comprehension. It begins by trying to interpret for pair in my_dict[letter] and cannot make any sense of this unless letter was already defined. If letter was previously defined as b (value left over from running a previous list comprehension) then it sets pair to the keys of my_dict['b']. It then looks at for letter in my_dict and sets letter to 'a' and to 'b'. It then tries to evaluate the first part, my_dict[letter][pair], but it's using the keys from b so this won't work when letter takes the value 'a'.

Below, I run your comprehension and get NameError, then run another comprehension which as a side-effect sets the value of letter, then I run your same comprehension again and get KeyError.

Python 2.6.9 
>>> my_dict = {
...     'a': {(1,2): True,
...           (1,3): False},
...     'b': {(1,4): True,
...           (2,3): False}
... }
>>> 
>>> [my_dict[letter][pair] for pair in my_dict[letter] for letter in my_dict]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'letter' is not defined
>>> letter
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'letter' is not defined
>>> [letter for letter in my_dict]
['a', 'b']
>>> letter
'b'
>>> [my_dict[letter][pair] for pair in my_dict[letter] for letter in my_dict]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: (2, 3)
>>>

Note in the above that the KeyError only happens after having (accidentally) set the value of letter. The first run produces NameError instead.

Ah, thanks a lot for the clarification on why I was getting a KeyError. It made it very confusing for me to debug. — elsherbini, Jan 19 '15 at 00:15

List Comprehension to Flatten a Dictionary of Dictionaries

2 Answers2