3

I recently had to debug some code that went something like this:

for key, group in itertools.groupby(csvGrid, lambda x: x[0]):
    value1 = sum(row[1] for row in group)
    value2 = sum(row[2] for row in group)
    results.append([key, value1, value2])

In every result set, value2 came out as 0. When I looked into it, I found that the first time the code iterated over group, it consumed it, so that the second time there were zero elements to iterate over.

Intuitively, I would expect group to be a list which can be iterated over an indefinite number of times, but instead it behaves like an iterator which can only be iterated once. Is there any good reason why this is the case?

Mason Wheeler
  • 82,511
  • 50
  • 270
  • 477

4 Answers4

7

itertools is an iterator library, and like just about everything else in the library, the itertools.groupby groups are iterators. There isn't a single function in all of itertools that returns a sequence.

The reasons the groupby groups are iterators are the same reasons everything else in itertools is an iterator:

  1. It's more memory efficient.
  2. The groups could be infinite.
  3. You can get results immediately instead of waiting for the whole group to be ready.

Additionally, the groups are iterators because you might only want the keys, in which case materializing the groups would be a waste.

itertools.groupby is not intended to be an exact match for any LINQ construct, SQL clause, or other thing that goes by the name "group by". Its grouping behavior is closer to an extension of Unix's uniq command than what LINQ or SQL do, although the fact that it makes groups means it's not an exact match for uniq either.

As an example of something you could do with itertools.groupby that you couldn't with the other tools I've named, here's a run-length encoder:

def runlengthencode(iterable):
    for key, group in groupby(iterable):
        yield (key, sum(1 for val in group))
user2357112
  • 260,549
  • 28
  • 431
  • 505
  • Considering how [itertools.groupby doesn't even work unless you sort your data set first](http://stackoverflow.com/a/7286/32914), and sorting can't be done on an infinite sequence, point 2 makes no sense. – Mason Wheeler Jan 15 '16 at 19:13
  • @MasonWheeler: The idea is that the input might be generated pre-sorted, perhaps because it's an order-preserving `imap` over a sorted sequence, or because you're streaming results from a database and the database ordered them for you, or because of some other property of the data source. If `itertools.groupby` couldn't operate on and return iterators, it probably wouldn't be in itertools at all. – user2357112 Jan 15 '16 at 19:18
  • Sure, I expect it to *operate on* iterators, sure, but *return* iterators? Not on a function like this! If [`itertools.groupby` is supposed to be equivalent to LINQ's `GroupBy()`,](http://sayspy.blogspot.com/2006/02/why-python-doesnt-need-something-like.html) I expect it to return [a grouping.](https://msdn.microsoft.com/library/bb344977%28v=vs.100%29.aspx) Eager evaluation is the only way to do a GroupBy operation correctly. – Mason Wheeler Jan 15 '16 at 19:23
  • @MasonWheeler: `itertools.groupby` is not intended to be equivalent to any LINQ construct. It predates LINQ, anyway, so it wouldn't be reasonable to expect it to imitate LINQ's design decisions. – user2357112 Jan 15 '16 at 19:31
2

Intuitively, I would expect group to be a list which can be iterated over an indefinite number of times, but instead it behaves like an iterator which can only be iterated once.

That's correct.

Is there any good reason why this is the case?

  1. It's potentially more memory efficient: you don't need to build an entire list first and then store it in memory, only to then iterate over it. Instead, you can process the elements as you iterate.

  2. It's potentially more CPU efficient: by not generating all data up front, e.g. by producing a list, you can bail out early: if you find a particular group which matches some predicate, you can stop iteration - no further work needs to be done.

The decision of whether you need all data and iterate it multiple times is not hardcoded by the callee but is left to the caller.

Frerich Raabe
  • 90,689
  • 19
  • 115
  • 207
1

From the docs

The returned group is itself an iterator that shares the underlying iterable with groupby(). Because the source is shared, when the groupby() object is advanced, the previous group is no longer visible. So, if that data is needed later, it should be stored as a list

Interestingly, if you don't consume g yourself, groupby will do it before returning the next iteration.

>>> def vals():
...     for i in range(10):
...         print(i)
...         yield i
... 
>>> for k,g in itertools.groupby(vals(), lambda x: x<5):
...     print('processing group')
... 
0
processing group
1
2
3
4
5
processing group
6
7
8
9
tdelaney
  • 73,364
  • 6
  • 83
  • 116
0

I got the same issue when trying to access a "groupby" returned iterator multiple times. Based on Python3 doc , it suggests transfer iterator to list , so that is can be accessed later.

Jia
  • 2,417
  • 1
  • 15
  • 25