2

I was reading e-satis' answer to What does the "yield" keyword do in Python?. And He said:

These iterables are handy because you can read them as much as you wish, but you store all the values in memory and this is not always what you want when you have a lot of values

which I don't really agree with. However I can't comment there.

Then comes this question: Do Python's iterables really store all values in memory?

I used to think so. But I changed my view since I saw Python's detailed documentation yesterday.

>>> import sys
>>> def gen(): 
...   n = 0                                                                                          
...   while n < 10:                                                                                 
...     yield n                                                                                     
...     n += 1
...
>>> a = [0,1,2,3,4,5,6,7,8,9]
>>> b = range(10)  # b is a range object, which is a iterable
>>> c = gen(10)    # c is a iterator, which is a iterable too
>>> sys.getsizeof(a)
144
>>> sys.getsizeof(b)
48
>>> sys.getsizeof(c)
72
>>> B = list(b)                                                                                     
>>> C = list(c)                                                                                     
>>> a
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> B
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> C
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> sys.getsizeof(B)                                                                                
200                                                                                                
>>> sys.getsizeof(C)                                                                           
160

We mistakenly think the iterable stores all values in memory, because we are used to get all values when using an iterable.

Am I right?

Community
  • 1
  • 1
kissg
  • 41
  • 5
  • 3
    By "these iterables" in that answer I suppose the poster meant those lists, not iterables in general. A list does store its values in memory, but an iterable necessarily doesn't have to. – Ilja Everilä Apr 14 '16 at 09:39
  • The paragraph you quoted refers to the iterables mentioned before it, not the ones mentioned after (like generators). – molbdnilo Apr 14 '16 at 09:46

1 Answers1

6

You stripped too much context when quoting from that answer. It was made specifically in the context of iterables like lists. Generators and the yield keyword are covered separately further down the answer.

Expanding your quote:

Everything you can use "for... in..." on is an iterable; lists, strings, files...

These iterables are handy because you can read them as much as you wish, but you store all the values in memory and this is not always what you want when you have a lot of values.

The claim is true for lists and string, they are stored in memory. It is however incorrect for files as you can iterate over a file without storing it all in memory.

A python iterable may or may not store the values in memory. It steps through the values one at a time and at each step it could pull the value from memory, create it from scratch, or read it from another source such as a file.

range() is a good example: in Python 2.x it generates all the values up front and stores them in memory, in Python 3.x it generates them as needed. Both return an iterable object.

Bhargav Rao
  • 50,140
  • 28
  • 121
  • 140
Duncan
  • 92,073
  • 11
  • 122
  • 156
  • Now I understand iterable more clearly. According to python's documentation: iterable is an object capable of returning its members one at a time. That's iterable's characteristic, whether it store all the values in memory or not. It may or may not store the values in memory, thank you~ – kissg Apr 14 '16 at 12:35