Python dictionaries are not allowed to change size but lists are?

Question

I think I just found a gotcha in Python.

Take a look at the following code:

a = {'a':1, 'b':2, 'c':3}

for i in a.values():
    if i == 2:
        a['d'] = 4
        
print(a)

This will result in a RuntimeError of:

Traceback (most recent call last):
  File "<string>", line 4, in <module>
RuntimeError: dictionary changed size during iteration

I come from a Java background so this makes sense that you can't actually modify data structures while you are iterating over them.

But this logic falls apart when you see the following code work fine for lists:

a = [1,2,3,4]

for i in iter(a):
    if i == 2:
        a.extend([5,6,7])
        
print(a)

The code outputs as expected:

[1, 2, 3, 4, 5, 6, 7]

Why does this work for lists but fail for dictionaries ?

This makes no sense.

I know that dict.values(), dict.items(), dict.keys() all return iterators BUT SO DO LISTS.

This code is using an iterator, right ?

a = [1,2,3,4]

for i in iter(a):
    if i == 2:
        a.extend([5,6,7])
        
print(a)

Correct me if I am wrong.

I did check this answer but I failed to get a concrete answer.

Why the special treatment for dictionaries?

Am I missing something very basic here?

`for x in a` is not an iterator, and any of these is not a part of the standard. In other implementations of Python you might get different results — Marat, Jul 23 '20 at 19:06
@Marat Check my code now. I am specifically using an iterator with a list. — ng.newbie, Jul 23 '20 at 19:11
the second part still holds, it is an undefined behavior. It is not an intentional part of the language design — Marat, Jul 23 '20 at 19:20
@Marat So all this is unintentional ? Nothing described here is standard ? That doesn't make a whole lot of sense. — ng.newbie, Jul 23 '20 at 19:21
"I know that dict.values(), dict.items(), dict.keys() all return iterators...." No, technically [they return views](https://docs.python.org/3/library/stdtypes.html#dict-views). — Brian McCutchon, Jul 23 '20 at 19:28
FWIW, `iter(a)` is not needed to reproduce the same behaviour. — wjandrea, Jul 23 '20 at 19:33
@wjandrea FWIW, My original question did not contain it. I added it after Marat said lists don't return iterators. — ng.newbie, Jul 23 '20 at 19:34
I believe that both Python and Java frown on the modification of iterated-over containers, but that neither language *guarantees* that it will be detected. However, the marginal is too narrow to fit... er, I mean I can't find this in the docs for either language right now, and it's bedtime here. :) — Ture Pålsson, Jul 23 '20 at 19:35
@TurePålsson Excatly. I even skimmed over the PEP for the iterators and couldn't find anything about this. — ng.newbie, Jul 23 '20 at 19:36
`a.values()` returns an iterable in Python3, not an iterator. By calling `iter(a)`, you're returning an iterable which is not allowed to change size. — Alex R., Jul 23 '20 at 21:52

score -1 · Answer 1 · answered Jul 23 '20 at 19:22

-1

You can easily avoid this by doing this

a = {'a' : 1, 'b' : 2, 'c' : 3}
for i in list(a.values()):
    if i == 2:
        a['d'] = 4
print(a)

As for why the different treatment I don't know, by but as long as you're not iterating through the dictionary itself it will work

answered Jul 23 '20 at 19:22

12ksins

307
1
12

I don't want to avoid it. I want to know why it is happening. I don't want a work-around I want a reason. The language design is inconsistent otherwise. – ng.newbie Jul 23 '20 at 19:23
@ng.newbie inconsistent language design is hardly unique to Python. I suspect the reason is that an iterator into a `dict` is tightly bound to the internal structure of that dict, and can't handle updates in real time. There's really nothing wrong with a work-around. – Mark Ransom Jul 23 '20 at 19:29
@MarkRansom Fully agree there is nothing wrong with the work-around but there has to be a concrete reason for this. Or else updates while iterating should be disabled on lists as well. – ng.newbie Jul 23 '20 at 19:31
3

@MarkRansom I suspect it's because dicts used to be unordered. I can't imagine adding/deleting items while iterating an unordered collection, that would be unpredictable/unreliable. But for lists I've done that many times (mostly appending). – Stefan Pochmann Jul 23 '20 at 19:48
I think @Stefan Pochmann is right however, if you're looking for good evidence of this I recommend checking other versions of python and see how they differ in this like IronPython *C# based* or Jython *java based* instead of CPython, or just check the source code of one of these versions and check for supposed errors in the dict class; here is the Cpython source code https://github.com/python/cpython and here's Jython's https://github.com/jython hope it helps – 12ksins Jul 23 '20 at 19:59
@StefanPochmann Wait, aren't lists also unordered ? I have seen no explicit ordering imposed on lists. I may be wrong though. – ng.newbie Jul 23 '20 at 20:03
@ng.newbie I think this is the most use full for your case https://github.com/python/cpython/blob/master/Objects/dictobject.c – 12ksins Jul 23 '20 at 20:05
2

@ng.newbie only sets and dicts are unordered however lists and tuples are ordered – 12ksins Jul 23 '20 at 20:07
1

@ng.newbie Lists unordered? I hope you're joking :-) – Stefan Pochmann Jul 23 '20 at 20:11

score -1 · Answer 2 · answered Jul 24 '20 at 14:37

-1

You have it backwards. The special treatment is not for dictionaries, the special treatment is for lists. That's because it's trivial to iterate through a changing list - you keep an integer that tells you the offset into the list. Every time you try to access an item through the iterator, you can check to see that it's still within the bounds of the list and generate an exception if it isn't.

>>> a = [1]
>>> x = 2
>>> for i in a:
    print(i)
    if x < 10:
        a[0:0] = [x]
        x += 1

1
1
1
1
1
1
1
1
1
>>> a
[9, 8, 7, 6, 5, 4, 3, 2, 1]

answered Jul 24 '20 at 14:37

Mark Ransom

299,747
42
398
622

1

I do see special treatment for dictionaries [here](https://github.com/python/cpython/blob/0dd98c2d00a75efbec19c2ed942923981bc06683/Objects/dictobject.c#L3743-L3748). What/where is the special treatment for lists? – Stefan Pochmann Jul 24 '20 at 15:15
@StefanPochmann what I meant by that was the general prohibition on modifying a sequence that you're iterating. That's the base case and anything that deviates from it is special. Obviously the construction of the data structure is going to dictate the mechanics of how you iterate it, so will require custom code. – Mark Ransom Jul 24 '20 at 15:55

Python dictionaries are not allowed to change size but lists are?

2 Answers2