6
def partition(n, iterable):
   p = izip_longest(*([iter(iterable)] * n))
   r = []
   for x in p:
       print(x) #I added this
       s = set(x)
       s.discard(None)
       r.append(list(s))
   return r

This is actually in a job posting on SO and being a newbie I thought it was interesting. So you get output like the following:

partition(5, L)
(1, 2, 3, 4, None)
Out[86]: [[1, 2, 3, 4]]

To me this is already confusing because I thought izip_longest(*([iter(iterable)] * n))would run the izip_longest function on a list of n identical iterators so I would have expected first an output of (1,1,1,1,1) and then an output of (2,2,2,2,2) and so on.

So short version of my question, is what's going on with this line:

 p = izip_longest(*([iter(iterable)] * n))

Parsing it I would have thought [iter(iterable)]*n creates a list of length n of identical iterables all pointing to the same thing - that's what it does on the command line, but that doesn't seem to be what it does here based on the output printed above.

Also I thought the * at the beginning ...longest(*... was there since the list is of unknown length but I don't think that entirely makes sense. What is that first * symbol doing inside the function call? Doesn't seem like it's simply indicating an unknown length list of arguments...

So at the end of the day I'm completely lost. Can someone walk me through this syntax?

Thanks a lot for any input!

Thanks for all the helpful answers, everyone. I am not sure if I am tacking on an answer or a question here, but it seems to me this list comprehension will do the same thing for lists and tuples (I realize iterators would also apply to dictionaries, custom classes, other things...)

[L[i*n:(i+1)*n] for i in range(int(ceil(len(L)/float(n)))) ]
sunny
  • 3,853
  • 5
  • 32
  • 62
  • 1
    Have you read [How does `zip(*[iter(s)]*n)` work in Python](http://stackoverflow.com/questions/2233204/how-does-zipitersn-work-in-python)? – Ashwini Chaudhary May 21 '15 at 16:42
  • it breaks a big list into N sized smaller lists .... filling None in the event that the list length is not divisible evenly by N ... the only sublist that should contain any `None` is the last list – Joran Beasley May 21 '15 at 16:44
  • Add a `print(list(p))` to see what `izip_longest()` did. – martineau May 21 '15 at 16:47
  • 1
    @AshwiniChaudhary thanks it was all about the first * - I thought that was only used in function definitions, so now I've learned a new related use. – sunny May 21 '15 at 16:55

2 Answers2

6

iter(my_list) converts a list into an iterable (that is one where elements are consumed as they are seen)

[my_iter]*5 creates a new list of [my_iter,my_iter,my_iter,my_iter,my_iter] in which all the my_iter's point to the same exact iterator

zip(*[my_iter,my_iter,my_iter,my_iter,my_iter])

is the same as 

zip(my_iter,my_iter,my_iter,my_iter,my_iter)

(the splat simply unpacks a list/tuple) which basically just returns a 5xlen(my_list)//5 2d list

you could simplify it with normal zip

#this method will have no `None` entries
new_list_partitioned = zip(*[iter(big_list)]*N) + [big_list[-(len(big_list)%N):],]
Joran Beasley
  • 110,522
  • 12
  • 160
  • 179
  • if you look at my question, that part I got. What I didn't get was why the * in front of [iter(... – sunny May 21 '15 at 16:52
6

Given:

>>> li
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20]

There is a common Python idiom of using zip in combination with iter and * operator to partition a list a flat list into a list of lists of n length:

>>> n=3
>>> zip(*([iter(li)] * n))
[(0, 1, 2), (3, 4, 5), (6, 7, 8), (9, 10, 11), (12, 13, 14), (15, 16, 17), (18, 19, 20)]

However, if n is not an even multiple of the overall length, the final list is truncated:

>>> n=4
>>> zip(*([iter(li)] * n))
[(0, 1, 2, 3), (4, 5, 6, 7), (8, 9, 10, 11), (12, 13, 14, 15), (16, 17, 18, 19)]

You can use izip_longest to use the complete list filled in with a selected value for the incomplete sub lists:

>>> list(izip_longest(*([iter(li)] * n)))
[(0, 1, 2, 3), (4, 5, 6, 7), (8, 9, 10, 11), (12, 13, 14, 15), (16, 17, 18, 19), (20, None, None, None)]
Community
  • 1
  • 1
dawg
  • 98,345
  • 23
  • 131
  • 206
  • good answer on explaining why he was getting None and why its using izip_longest :) +1 – Joran Beasley May 21 '15 at 16:55
  • 1
    It should be noted that as a consequence of the way the extra `None`s are being removed, that iterables containing `None` are unsupported by this function. For example, `partition(2, [1,2,None,4,5])` gives `[[1, 2], [4], [5]]`. It would have been more robust to use a custom made sentinel value, I think. – jpmc26 May 21 '15 at 21:29