Why don't Python dictionaries treat keys independently in this script?

Question

I'm expecting my frustration to be overridden with some enlightenment - here's a minimal version of the script to demonstrate the problem:

First I create a dictionary:

dic  = {
    'foo':{}, 
    'bar':{}
    }

Then we instantiate a template dictionary that can be iteratively appended to keys of dic:

appendic= {  
    'is':'',       #  '' is a terminal value to be replaced later
}

So here we append appendic to each of the keys in dic:

dic['foo'] = appendic
dic['bar'] = appendic

Now we replace the terminal values, '', with something meaningful:

dic['foo']['is'] = 'foo'
dic['bar']['is'] = 'bar'

At this point, my intuition tells me that if we call:

print(dic['foo']['is']) we get 'foo'

But instead Python returns 'bar' ... to my un-trained mind that is counter-intuitive.

Questions:

How can I tell Python to keep the keys of dic independent?
Why is this the default behaviour? What use cases does this have?

score 9 · Accepted Answer · answered May 08 '14 at 15:56

9

When you assign a appendic to two different keys, Python doesn't make a copy. It assigns a reference instead.

As a result, both dic['please_make_me_Foo'] and dic['dont_make_him_Bar'] refer to the same object. These are not separate dictionaries, they are both the same object, the one appendic also references to.

If you expected these to be separate dictionaries, create a copy of appendic instead. The dict.copy() method creates a shallow copy of a dictionary:

dic['please_make_me_Foo']= appendic.copy()
dic['dont_make_him_Bar'] = appendic.copy()

Shallow means that a new dictionary is created and all references to keys and values contained are copied over.

If appendic itself contains values that are also dictionaries, these would not be copied. The new copy and appendic would both refer to the same values. In most cases, that's not a problem because most primitive values (strings, integers, etc.) are immutable, and you never notice references are shared as you replace such values with new ones.

answered May 08 '14 at 15:56

Martijn Pieters

1,048,767
296
4,058
3,343

Cheers again @Martijn Pieters, would you happen to know cases where this can be useful? (I actually wanted to know a little of why the default behaviour acts the way it does) – hello_there_andy May 08 '14 at 16:16
@hello_there_andy: Loads of places where this is useful; the whole of Python is built on top of objects being referenceable from multiple locations. – Martijn Pieters May 08 '14 at 16:17
1

@hello_there_andy: If Python made a copy of the object each time you stored a reference to it, you'd need a lot more memory to run your code, for starters. – Martijn Pieters May 08 '14 at 16:18
1

This has to be a duplicate of at least a hundred questions, which you should know because you probably answered half of them. Why not close this or at least the next one as a duplicate instead? I'll even help you search for a suitable one once I'm back at a pc and not on mobile... – l4mpi May 08 '14 at 16:21
1

@hello_there_andy: This is just how assignment in Python works. Since you were suprised, I suggest reading [Facts and myths about Python names and values](http://nedbatchelder.com/text/names.html) by Ned Batchelder. – Steven Rumbalski May 08 '14 at 16:22
2

@l4mpi: I'd love to find a suitable one. We really need a more generic 'halp my dictionary / list / set / custom class is being shared' post somewhere as a dupe target. – Martijn Pieters May 08 '14 at 16:23
I did do some searching for dupes, but there's many unhelpful titles, I originally typed "why does the python dict behave like this" and seeing so many of those made me try a little harder on the title – hello_there_andy May 08 '14 at 16:44
3

@hello_there_andy: we didn't expect you to find the dupe; when you don't understand yet that Python stores references to objects the behaviour can be hard to search for, sure. What l4mpi is referring to is that he feels I should know better and have found you a dupe. :-) – Martijn Pieters May 08 '14 at 16:46
1

@hello_there_andy: if a suitable dupe is found, don't worry. Your question will then serve as a signpost to the other question, for future searchers to find. – Martijn Pieters May 08 '14 at 16:46
I did some searching, sadly I couldn't find a good generic question. [This question](http://stackoverflow.com/q/2465921) would fit here (among a few others), but for the general case a question/answer should at least explain immutability and give a short overview of references (and how it doesn't matter if you assign it to a variable/function argument/dict entry/etc). Creating a canonical question covering the most common cases would probably be a good idea. There's also the reverse where people ask why immutable objects don't behave this way but that's probably too broad to cover as well... – l4mpi May 08 '14 at 18:11

score 4 · Answer 2 · answered May 08 '14 at 15:56

You make a dict:

appendic= {  
    'Python_made_me':''
}

Add it to your other dict twice

dic['please_make_me_Foo']= appendic
dic['dont_make_him_Bar'] = appendic

And set the single dict's Python_made_me value twice

dic['please_make_me_Foo']['Python_made_me'] = 'Foo'
dic['dont_make_him_Bar']['Python_made_me']  = 'Bar'

But because they're the same dict, the second line overwrites the first

If you need to copy it, you need to use the copy method:

dic['please_make_me_Foo']= appendic.copy()
dic['dont_make_him_Bar'] = appendic.copy()

score 3 · Answer 3 · answered May 08 '14 at 16:02

ok, I'm just going to write this as a complement to the other answers. When you manipulate a dictionary, you manipulate the reference to an instance, which is the root cause of your mistake. Using hex(id(foo)) you get the memory address of foo, so let's show the address of d instance in the following example to make that tangible:

>>> hex(id(d))
'0x10bd95e60'
>>> hex(id(e[1]))
'0x10bd95e60'
>>> hex(id(f[1]))
'0x10bd95e60'

so if you add or remove values from e[1], you're actually changing the same instance as the one pointed by d, and as a dictionary is mutable, i.e. you can change values within.

Now you're wondering why that does not happen when you're handling integers? Because, in fact it does, it's just that integers are not mutable:

>>> i = 1
>>> hex(id(i))
'0x10ba51e90'
>>> j = i
>>> hex(id(j))
'0x10ba51e90'
>>> i = 2
>>> hex(id(i))
'0x10ba51eb0'

i.e. i is pointing to another place in the memory.

It's possible to create a mutable integer though, by using a class:

>>> class Integer:
...   def __init__(self, i):
...     self.i = i
... 
>>> i = Integer(2)
>>> hex(id(i))
'0x10bd9b410'
>>> j = i
>>> hex(id(j))
'0x10bd9b410'
>>> j.i = 2
>>> i.i
2
>>> hex(id(i))
'0x10bd9b410'

In order to create a new instance of the same dictionary, you need to use the copy() member of a dict:

>>> hex(id(d))
'0x10bd95e60'
>>> w = d.copy()
>>> x = d.copy()
>>> y = d.copy()
>>> hex(id(w))
'0x10bd96128'
>>> hex(id(x))
'0x10bd95f80'
>>> hex(id(y))
'0x10bd96098'

score 2 · Answer 4 · answered May 08 '14 at 15:56

dic['please_make_me_Foo']= appendic
dic['dont_make_him_Bar'] = appendic

appendic is an object - you are assigning a reference to the same object to both keys in dic. So when you change one, you change both.

Try this instead:

dic['please_make_me_Foo']= appendic.copy()
dic['dont_make_him_Bar'] = appendic.copy()

Why don't Python dictionaries treat keys independently in this script?

4 Answers4