3

Possible Duplicate:
Is there a map without result in python?

I often come to a situation in my programs when I want to quickly/efficiently call an in-place method on each of the items contained by an iterable. (Quickly meaning the overhead of a for loop is unacceptable). A good example would be a list of sprites when I want to call draw() on each of the Sprite objects.

I know I can do something like this:

[sprite.draw() for sprite in sprite_list]

But I feel like the list comprehension is misused since I'm not using the returned list. The same goes for the map function. Stone me for premature optimization, but I also don't want the overhead of the return value.

What I want to know is if there's a method in Python that lets me do what I just explained, perhaps like the hypothetical function I suggest below:

do_all(sprite_list, draw)

Community
  • 1
  • 1
Thane Brimhall
  • 9,256
  • 7
  • 36
  • 50
  • 17
    *The overhead of a `for` loop is unacceptable?* That is total nonsense! Total, total nonsense! – Dietrich Epp Dec 08 '12 at 00:38
  • 1
    You could just write `any(sprite.draw() for sprite in sprite_list)` if the function doesn't return anything. If it does, you might be able use `all()` instead depending on what it returns. – martineau Dec 08 '12 at 01:03
  • 4
    The cost of a `for` loop is dwarfed by the cost of your function call. – James Thiele Dec 08 '12 at 01:08
  • @martineau: That generates byte code that is basically identical to the `for` loop, except with the extra overhead of creating a generator object and looping through it in another function. – Dietrich Epp Dec 08 '12 at 01:17
  • @DietrichEpp: It's less typing and the extra overhead of the generator object is likely less than that of a list object that also has to be appended-to multiple times. – martineau Dec 08 '12 at 01:28
  • 1
    @martineau: Less typing than `for sprite in sprite_list: sprite.draw()` ? It looks like more typing... – Dietrich Epp Dec 08 '12 at 01:50
  • @DietrichEpp: It's less typing than writing a `do_all()` function and doesn't allocate and manipulate a `list` object. My original comment was addressing items in the OP's question. – martineau Dec 08 '12 at 02:25
  • 1
    @martineau: It's less typing if you do it once; if it's something you do all the time (as in "I often come to a situation…"), it's less typing to implement the `do_all` function once. Also, if there really were a performance implication here, a `do_all` gives you a central place, so if you find a faster way to do it, you can just change it once. – abarnert Dec 09 '12 at 22:49
  • @DietrichEpp: also list comprehension can be easily replaced by a simple 'for', but Guido thought it was good to have it in Python... the OP simply asks for a similar feature. – Don Sep 06 '13 at 15:09

2 Answers2

10

You can always write your own do_all function:

def do_all(iterable, func):
    for i in iter(iterable):
       func(i)

Then call it whenever you want.

There is really no performance problem with using an explicit for loop.

There is a performance problem with using a list comprehension or map, but only in building the list of results. Obviously, iterating over 500M items will be a lot slower if you have to build up a 500M list along the way.

It's worth pointing out here that this is almost certainly not going to arise for things like drawing a list of sprites. You don't have 500M sprites to draw. And if you do, it'll probably take a lot longer than creating a list of 500M copies of None. And in most plausible cases where you do need to do the same very simple thing to 500M objects, there are better solutions, like switching to numpy. But there are some conceivable cases where this could arise.

The easy way around that is to use a generator expression or itertools.imap (or, in Python 3, just map) and then dispose of the values by writing a dispose function. One possibility:

def dispose(iterator):
    for i in iterator:
        pass

Then:

dispose(itertools.imap(Sprite.draw, sprite_list))

You could even define do_all as:

def do_all(iterable, func):
    dispose(itertools.imap(func, iterable))

If you're doing this for clarity or simplicity, I think it's misguided. The for loop version is perfectly easy to read, and this version looks like you're trying to write Haskell with the wrong function names and syntax.

If you're doing it for performance… well, if there were ever a real-life performance situation where this mattered (which doesn't seem very likely), you'd probably want to play with a bunch of different potential implementations of dispose, and possibly move the dispose back into the do_all to avoid the extra function call, and maybe even implement the whole thing in C (borrowing the fast-iteration code from the stdlib's itertools.c).

Or, better, pip install more-itertools, then use more_itertools.consume. For what it's worth, the current version just does collections.deque(iterator, maxlen=0), and in a test against a custom C implementation, it's less than 1% slower (except for very tiny iterators—the cutoff is 19 on my system), so it's probably not worth implementing in C. But if someone does, or if some future Python (or PyPy) provides a faster way to implement it, chances are it'll be added into more-itertools before you find out about it and change your code.

abarnert
  • 354,177
  • 51
  • 601
  • 671
  • The `itertools.dropwhile` version will do at least two function calls per iteration. There's no way it's won't be worse than a `for` loop. – Dietrich Epp Dec 08 '12 at 00:47
  • @DietrichEpp: The iteration itself could be significantly faster, because the C implementation can short-circuit some cases, and doesn't need to create extra temp variables/references, etc.. You're right that this would only be true if the predicate itself were a C function… But really, if this _were_ an issue, you'd probably want to write a `my_itertools.dispose` in C (not hard based on the `itertools` source). Fortunately, this is almost never actually an issue. – abarnert Dec 08 '12 at 00:52
  • Pure speculation. Writing an outer loop in C is a waste of effort, if the inner function is written in Python. – Dietrich Epp Dec 08 '12 at 01:01
  • 1
    @DietrichEpp: Yes, as I said, the predicate has to be a C function, and the code that calls that predicate also has to be in C (like most of `itertools` is). And of course it's speculation; the whole point is that you shouldn't guess what might be fastest—if it really matters, try different implementations and profile them; if it doesn't really matter, don't worry about it, and write what's readable. – abarnert Dec 08 '12 at 01:06
  • 2
    Not that anyone cares, but I think I've found the fastest way to consume an iterator without doing anything. See the answer. – abarnert Dec 09 '12 at 22:51
  • I care! Thanks for the more-itertools.consume tip. I wonder if it still uses the deque with maxlen 0 trick or if there has been some improvement in the implementation since then. – Davos May 10 '18 at 07:56
7

Assuming sprite_list is a list of Sprite objects, you can do:

map(Sprite.draw, sprite_list)

This will call Sprite.draw() on each item in sprite_list which is essentially the same as the list comprehension you posted. If you don't want to create a list, you can just use a for loop:

for sprite in sprite_list:
    sprite.draw()
Matt
  • 3,651
  • 3
  • 16
  • 35
  • 5
    The problem with map is that it ALSO returns a list of the results, which I specifically do not need/want. I mentioned it as one of the ways I know work, but aren't the ideal. – Thane Brimhall Dec 08 '12 at 00:38
  • how bout filter instead ... http://stackoverflow.com/questions/1080026/is-there-a-map-without-result-in-python ... `filter(my_func,my_items)` – Joran Beasley Dec 08 '12 at 00:43
  • @JoranBeasley: That seems like a horrible misuse of `filter`, not to mention that it only works if you're sure `Sprite.draw` actually always returns `None` (or something falsey) rather than just "something I want to ignore". But yeah, if the performance ever really mattered, it would probably be something else to try. – abarnert Dec 08 '12 at 00:48
  • well I was really more pointing out that this question has been asked and it appears the answer is no ... I would expect map to actually be slightly faster (strictly as a guess) – Joran Beasley Dec 08 '12 at 00:58
  • Why not just discard the result, @ThaneBrimhall? – hd1 Dec 08 '12 at 01:05
  • 1
    @hd1: Presumably because the result is a length-N list, and if N is huge, building it just to discard it may mean your program ends up spending more time swapping memory around than doing real work? Obviously that's _not_ an issue in the OP's case (nobody has a billion sprites to draw, and if he did, drawing all of those sprites would still take longer than maintaining a list of a billion `None`s), but there are cases where it _can_ be an issue. – abarnert Dec 08 '12 at 01:10
  • @abarnert: Correct, sprites were just an example, but I have other use cases which go into the tens of thousands of large objects. – Thane Brimhall Dec 12 '12 at 23:00
  • A note from the future: in python3 `map` returns a mapping object _iterator_. So you'd need to do `list(map(Sprite.draw, sprite_list))` to actually call `draw` and get the equivalent python2 behaviour. – Hamish Aug 03 '17 at 05:23