filter object becomes empty after iteration?

Question

I'm learning how to use the filter function.

This is the code I've written:

people = [{'name': 'Mary', 'height': 160},
          {'name': 'Isla', 'height': 80},
          {'name': 'Sam'}]

people2 = filter(lambda x: "height" in x, people)

As you can see what I'm trying to do is to remove all the dictionaries that don't contain the 'height' key.

The code works properly, in fact if I do:

print(list(people2))

I get:

[{'name': 'Mary', 'height': 160}, {'name': 'Isla', 'height': 80}]

The problem is that if I do it twice:

print(list(people2))
print(list(people2))

the second time, I get an empty list.

Can you explain me why?

Related: https://stackoverflow.com/questions/21715268/list-returned-by-map-function-disappears-after-one-use , https://stackoverflow.com/questions/40960036/cannot-use-list-more-than-once-on-a-map-object — Mark Dickinson, Jun 07 '17 at 18:25

score 23 · Accepted Answer · answered Jun 07 '17 at 18:26

23

This is a classic python3 doh!.

A filter is a special iterable object you can iterate over. However, much like a generator, you can iterate over it only once. So, by calling list(people2), you are iterating over each element of the filter object to generate the list. At this point, you've reached the end of the iterable and nothing more to return.

So, when you call list(people2) again, you get an empty list.

Demo:

>>> l = range(10)
>>> k = filter(lambda x: x > 5, l)
>>> list(k)
[6, 7, 8, 9]
>>> list(k)
[]

I should mention that with python2, filter returns a list, so you don't run into this issue. The problem arises when you bring py3's lazy evaluation into the picture.

answered Jun 07 '17 at 18:26

cs95

379,657
97
704
746

oh, ok, so if I want to use it more than once, I just have to save it on another variable, like *people3 = list(people2)*, right? – L'ultimo Jun 07 '17 at 18:40
@L'ultimo Yep, you got it. – cs95 Jun 07 '17 at 18:40
Ok, really thank you! :) – L'ultimo Jun 07 '17 at 18:41
(Or even better I can directly do *people2 = filter(lambda x: "height" in x, people)*) – L'ultimo Jun 07 '17 at 18:43
You'll need to do `people2 = list(filter(lambda x: "height" in x, people))` but yes. – cs95 Jun 07 '17 at 18:44
4

I don't understand the logic for this. filter and map are special objects, can they not have logics for reseting it back to the start after it's been parsed? You can still have lazy evaluation AND also be usable multiple times. Every time __iter__ is called, return a new lazy iterator from the start. – Ehsan Kia Nov 19 '18 at 17:22
This caveat should be pointed out clearly in tutorials like w3schools. It's far from obvious. Is there a reason for this behavior? – Fred Jul 22 '22 at 14:49

score 1 · Answer 2 · answered Jun 07 '17 at 18:27

It's because what filter really turns is an iterator. This iterator doesn't really do anything until you start to use it's results, in this case when you cast it to a list. people2 is this thing that's ready to filter the list of people, then when list is called on it, it iterates through the list of people and delivers the filtered result. Now that iterator is done, there's nothing left for it to iterate over, so when you call list on it a second time, there's nothing there.

Read this for some more details - Lazy evaluation python

filter object becomes empty after iteration?

2 Answers2

Linked