Weird unpacking in list comprehension

Question

I was watching a lecture from David Beazley. At minute 23:20 he does some "magic" with unpacking that I am having hard time understanding.

The "magic line" is

fail = [ { **row, 'DBA Name': row['DBA Name'].replace("'",'').upper() } for row in fail ]

I have searched for similar examples but I couldn't find any. Can you explain what is going on in this code? Can you point me to some similar examples?

@JimFasarakisHilliard: the other post is the canonical 'what does `**` mean post, and has an answer specifically covering this case. — Martijn Pieters, Aug 18 '17 at 15:03
I'm okay with linking both, but the focus of the parameters question is very different, and you don't actually see *anything* about PEP 448 unpacking generalizations until nine answers and *many* pages of scrolling down (and that answer isn't even a good answer to the OP's question). [asterisk in tuple, list and set definitions, double asterisk in dict definition](https://stackoverflow.com/a/36908/364696) is a much better target for the PEP 448 specific unpacking generalizations; the question and top answer are both about what the OP here is asking about. — ShadowRanger, Mar 22 '19 at 19:50

score 3 · Accepted Answer · answered Aug 18 '17 at 14:16

3

The snippet is unpacking an already existing mapping row in a dictionary literal while adding a new element. A simplified example demonstrating this:

>>> r = {'a':1, 'b':2}    
>>> {**r, 'Spam': 20}
{'Spam': 20, 'a': 1, 'b': 2}

This unpacking is only available in Pythons >= 3.5 as introduced with PEP 448; in previous versions it is a SyntaxError.

answered Aug 18 '17 at 14:16

Dimitris Fasarakis Hilliard

150,925
31
268
253

I didn't know that you can add a new element while unpacking. Is this in the official documentation? I can't find it :/ – alec_djinn Aug 18 '17 at 14:21
@alec_djinn yes, PEP 448 describes it and so does the language reference, take a look at [Dictionary displays](https://docs.python.org/3/reference/expressions.html#dictionary-displays) (see the sentence on `**`). – Dimitris Fasarakis Hilliard Aug 18 '17 at 14:25
Thanks! I missed the Dictionary displays paragraph. – alec_djinn Aug 18 '17 at 14:27
@alec_djinn: The idea is to make unpacking something that just dumps additional definitions into a "normal" `dict` literal. It's not "adding a new element", because that assumes the new `dict` is directly tied to the old one, and that there is one and only one old `dict` that it's based on. But you could unpack multiple `dict`s, with key/value pairs before, in between or after them, and it follows the same rules as regular `dict` literals (order is based on when a key is *first* seen, but the value comes from the *last* appearance of a key). – ShadowRanger Mar 22 '19 at 19:46

score 1 · Answer 2 · answered Aug 18 '17 at 14:50

The {**row} just unpacks the dictionary (which is allowed in literals in python-3.5+):

>>> row = {'DBA Name': "make 'this' now", 'b': 2}
>>> {**row}
{'DBA Name': "make 'this' now", 'b': 2}

Essentially this just makes a copy of row because you unpack a dictionary into a dictionary.

The second part is normal dictionary literal syntax:

>>> {'DBA Name': row['DBA Name'].replace("'",'').upper() }
{'DBA Name': 'MAKE THIS NOW'}

The "magic" bit is that dictionaries map unique keys to some value, so this is essentially a copy the row dictionary and replaces the 'DBA Name' key with the new value. That works because literals are interpreted from left to right:

>>> { **row, 'DBA Name': row['DBA Name'].replace("'",'').upper() }
{'DBA Name': 'MAKE THIS NOW', 'b': 2}

Normally you would just create a copy of the dictionary and replace the key:

>>> newrow = row.copy()
>>> newrow['DBA Name'] = row['DBA Name'].replace("'",'').upper()

But that doesn't work in comprehensions (assignment inside comprehensions is a SyntaxError) so you need some "magic" (or invoke special methods).

But you could also do it with a nested comprehension (it's slower but maybe less magic):

[        k: v.replace("'",'').upper() if key == 'DBA Name' else v 
 for row in fail 
     for k, v in row.items()]

The indentation may seem a bit off but I find it easier to visualize this way, you could also use just one line:

[k: v.replace("'",'').upper() if key == 'DBA Name' else v for row in fail for k, v in row.items()]

Weird unpacking in list comprehension

2 Answers2