5

The best answer in What is the most “pythonic” way to iterate over a list in chunks? using the the function izip_longest to chunk a list. But I cannot understand it.

def grouper(iterable, n, fillvalue=None):
    args = [iter(iterable)] * n
    return izip_longest(*args, fillvalue=fillvalue)

for item in grouper(range(10), 4):
    print list(item)

I run the code above, then the chunked lists is created: 

[1 ,2, 3, 4]
[5, 6, 7, 8]
[9, 10, None, None]

I tried to run it step by step:

In [1]: args = [iter(range(10))] * 4

In [2]: args
Out[2]: 
[<listiterator at 0x1ad7610>,
 <listiterator at 0x1ad7610>,
 <listiterator at 0x1ad7610>,
 <listiterator at 0x1ad7610>]

A list is created by the same iterator. I know the function izip_longest is implemented to generate pairs of lists. How is the iterator transformed to the chunked lists by izip_longest? Thanks.

Community
  • 1
  • 1
zfz
  • 1,597
  • 1
  • 22
  • 45

1 Answers1

7

The grouper function just zips the original iterable with offset versions of itself. Using [iter(iterable)] * n creates a list with n references to the same iterator. These are not independent copies; they are all references to the same object, so advancing one advances them all. Here's a simple example:

>>> x = [1, 2, 3]
>>> a, b = [iter(x)] * 2
>>> next(a)
1
>>> next(b)
2

izip_longest, like zip, takes one element at a time from each iterator. So first it grabs the first element from the first iterable in args, which will be the first element of the original iterable. But when it grabs this element, it advances all the iterators, because they are all linked. So when izip_longest goes to get an element from the next iterator, it will get the second element from the original iterable. It goes on like this; every time it grabs an element from one iterator, it advances all of them, so that the item it grabs from the next iterator will be the next item from the original iterable.

BrenBarn
  • 242,874
  • 37
  • 412
  • 384
  • Thanks. I lookup the source code of 'izip_longest', https://docs.python.org/2/library/itertools.html#itertools.izip_longest, still confused about the line 'iterators = [chain(it, sentinel(), fillers) for it in args]', How does the iterator in args chained? – zfz Mar 04 '15 at 06:31
  • @zfz: If you want to know about how `izip_longest` itself works internally, you should ask a separate question about that. – BrenBarn Mar 04 '15 at 06:41