When and why does the interpreter unravel by assuming same length sublists?

Question

I'm impressed by and enjoy the fact that a simple Python for statement can easily unravel a list of lists, without the need for numpy.unravel or an equivalent flatten function. However, the trade-off is now that I can't access elements of a list like this:

for a,b,c in [[5],[6],[7]]:
     print(str(a),str(b),str(c))
... 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: not enough values to unpack (expected 3, got 1)

and instead, this works, up until the length-1 [5]:

for a,b,c in [[1,2,3],[4,5,6],[7,8,9],[0,0,0], [5]]:
     print(a,b,c)

1 2 3
4 5 6
7 8 9
0 0 0
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: not enough values to unpack (expected 3, got 1)

Logically, it doesn't make sense to assume that a list would have a fixed number of elements. How come then, Python allows us to assume that a list of lists would always have the same number of elements?

I'd like to be aware of what Python expects, because I want to anticipate wrongly formatted lists/sublists.

I've poked around Python documentation and Stackoverflow, but haven't found the reasoning or how the interpreter is doing this.

My guess is that flattening same-length arrays is such a common occurrence (e.g. machine learning dimensionality reduction, matrix transformations, etc.), that there's utility in providing this feature at the trade-off of being unable to do what I've tried above.

Answer 1

The interpreter always assumes the length is matching when making an unpacking assignment, and just crashes with ValueError if it doesn't match. A for-loop is actually very similar to a kind of "repeated assignment statement", with the LHS being the free variable(s) of the loop and the RHS being an iterable container yielding the successive value(s) to use in each step of the iteration.

One assignment per iteration, made at the beginning of the loop body - in your case, it's an unpacking assignment, which binds multiple names.

So, in order to be properly equivalent to the second example, your first example which was:

for a,b,c in [[5],[6],[7]]:
    ...

should have been written instead:

for a, in [[5],[6],[7]]:
    ...

There is no "anticipation", and there can't be because (in the general case) you may be iterating over anything, e.g. data streaming in from a socket.

In order to fully grasp how for-loop flow works, the analogy with assignment statements is very useful. Anything that you can use on the left hand side of an assignment statement, you can use as the target in a for-loop. For example, this is equivalent to setting d[1] = 2 etc in a dict - and should make same result as dict(RHS):

>>> d = {}
>>> for k, d[k] in [[1, 2], [3, 4]]: 
...     pass 
...
>>> d
{1: 2, 3: 4}

It's just a bunch of assignments, in a well-defined order.

Answer 2

Python doesn't know, you just told it to expect three elements by unpacking to three names. The ValueError says "you told us three, but we found a sub-iterable that didn't have three elements, and we don't know what to do".

Python isn't really doing anything special to implement this; aside from special cases for built-in types like tuple (and probably list), the implementation is just to iterate the sub-iterable the expected number of times and dump all the values found on the interpreter stack, then store them to the provided names. It also tries to iterate one more time (expecting StopIteration) so you don't silently ignore extra values.

For limited cases, you can be flexible by having one of the unpack names preceded with a *, so you capture all the "didn't fit" elements into that name (as a list). That lets you set a minimum number of elements while allowing more, e.g. if you really only need the first element from your second example, you could do:

for a, *_ in [[1,2,3],[4,5,6],[7,8,9],[0,0,0], [5]]:
    print(a,b,c)

where _ is just a name that, by convention, means "I don't actually care about this value, but I needed a placeholder name".

Another example would be when you want the first and last element, but otherwise don't care about the middle:

for first, *middle, last in myiterable:
    ...

But otherwise, if you need to handle variable length iterables, don't unpack, just store to a single name and iterate that name manually in whatever way makes sense to your program logic.

Answer 3

Python does not assume same length lists because this is not only for lists.

When you iterate for a,b,c in [[1,2,3],[4,5,6],[7,8,9],[0,0,0], [5]] what is happening is that python returns a iterator that will iterate(return) each list values.

So that for is equivalent with:

l = [[1,2,3],[4,5,6],[7,8,9],[0,0,0], [5]]

l_iter = iter(l)

a,b,c = next(l_iter)

next(l_iter) will return each element from the list until it will raise a StopIteration execption according to the python iteration protocol.

This means:

a,b,c = [1,2,3]
a,b,c = [4,5,6]
a,b,c = [7,8,9]
a,b,c = [0,0,0]
a,b,c = [5]

As you can see now python can't unpack [5] into a,b,c as there is only one value.