How returning a map object by map() function helps better memory management and performance

Question

With Python 3+, map() function returns a map object (<map object at 0x01E7E150>) instead of list, when I tried to figure out the reason then some posts (like this) suggests that it helps in better memory management and performance.

Since I am new to Python so I really couldn't fully understand "HOW" it can help better memory management and performance. Best I could think is - suppose instead of returning the map object if map() had returned the list object then I need not to convert the map object to list like list(map(.....)), but since map object is returned so anyways in the very next step I have to call list() function.

I know an argument could be that you can directly loop through the map object but then at that time as well memory has to be allocated and function passed to map() has to be evaluated then how does it make difference if I do it now or one step later?

UPDATE: It is suggested that this could be a possible duplicate of this but I don't that answer is comprehensively answering by question about "memory management" and "performance" benefits.

Possible duplicate of [Why does map return a map object instead of a list in Python 3?](https://stackoverflow.com/questions/40015439/why-does-map-return-a-map-object-instead-of-a-list-in-python-3) — Chris_Rands, Mar 22 '18 at 18:33
Roughly, if you loop over the Py3 `map` object then you only hold in memory each item of the iterable at any one time, rather than the entire iterable (like with a `list`); see my suggested dupe for some other details/differences — Chris_Rands, Mar 22 '18 at 18:40
@Chris_Rands Yes, I did saw your dupe suggestion, but I don't that answer is comprehensively answering by question about "memory management" and "performance" benefits. — hagrawal7777, Mar 22 '18 at 18:41
I don't understand what you don't understand, do you understand how iterators/generators work generally in Python? — Chris_Rands, Mar 22 '18 at 18:43
Let's say you `map` a function on 1mio values. In a list comprehension, you would have to store all 1mio results in memory. If you iterate a `map` object in Python3, you can get the result save it to a file and move onto the next. *You never have to hold all 1mio results in memory.* Lower memory usage often means performance improvement. — jpp, Mar 22 '18 at 18:43
"in the very next step I have to call `list()` function" - frequently, you don't. The cases where you don't are why they made the change. — user2357112, Mar 22 '18 at 18:44

score 0 · Answer 1 · answered Mar 22 '18 at 19:33

Python 3's map returns an iterator (the map object you see in your output). This iterator is lazy, it only does the work of applying the function to one value from the input iterator at a time, in order to produce one output value. This is convenient when the input is also a lazy iterator, and the output can be consumed one value at a time.

Consider this code:

with open(some_file) as f_in, open(some_file + ".out", "w") as f_out:
    f_out.writelines(map(line_transform_func, f_in))

This opens two files, one for reading and one for writing. It then reads in lines from the input file, performs some kind of transform on them, then writes them to the output file. Since files are lazy, there's no need for the whole contents of the input file to be read into memory at once. Instead, you only need enough memory to read one line at a time. Similarly, since map returns a lazy iterator, the transformed lines will be produced one line at a time, and the writelines method can write each one out to disk before requesting the next one. This is very important if the files are huge (multiple gigabytes, maybe more than can fit in memory at one time).

Another advantage of map being a lazy iterator is that it doesn't need to run its function on all the input if you stop iterating on the output early. For example, lets say you had an expensive computation to do on a large input sequence, but you only need to do it until you find the first output that meets some criteria. You could use map and a loop with a break statement to stop iterating as soon as you found the value you needed:

for value in map(some_expensive_operation, input_values):
    if some_criteria(value):
        break

If map produced a list, it would need to apply the expensive operation to all the values from the input up front, before the iteration could happen.

(Note that I would not normally write like either of these examples using map at all, I'd just write an explicit for loop and apply the function myself, but map works fine too.)

If you're not familiar with Python iterators, I strongly recommend that you read up on them. They're a big part of how Python works, and understanding them (and how to create and use new lazy ones, e.g. with generator functions) is a big part of becoming a more skilled Python programmer.

How returning a map object by map() function helps better memory management and performance

1 Answers1