what's the trick behind python's any() function?

Question

One of the pycon2011 talks shared this any() function trick; the explanation was that the loop is in C.

Could someone explain more about it? What is the trick behind it and is there any other use cases?

>>> import itertools, hashlib, time
>>> _md5 = hashlib.md5()
>>> def run():
...   for i in itertools.repeat('foo', 10000000):
...     _md5.update(i)
... 
>>> a = time.time(); run(); time.time() -a
3.9815599918365479
>>> _md5 = hashlib.md5()
>>> def run():
...   any(itertools.imap(_md5.update, itertools.repeat('foo', 10000000)))
... 
>>> a = time.time(); run(); time.time() -a
2.1475138664245605
>>>

http://blip.tv/pycon-us-videos-2009-2010-2011/pycon-2011-how-dropbox-did-it-and-how-python-helped-4896698 — alexband, Oct 19 '11 at 03:11
Possibly duplicate question http://stackoverflow.com/questions/2012611/any-function-in-python-with-a-callback — Shashi, Oct 19 '11 at 03:51
you don't need `any`, in fact `filter(_md5.update, itertools.repeat('foo', 10000000))` would be even faster. — tomasz, Oct 21 '11 at 00:01
@alexband, it's not faster than `any` per se, but you don't need `imap`, hence less operations required. — tomasz, Oct 21 '11 at 09:12

score 4 · Answer 1 · edited Oct 20 '11 at 18:11

itertools.imap creates a lazy list which contains the function to be evaluated (md5) and it's argument ('foo' string). The md5 calls are not evaluated at this point but prepared along with their arguments (I think they are called thunks). When you then pass this iterator to any function, it goes through all the elements evaluating them. This happens faster than explicit Python evaluation from the first program, because any is implemented in C and everything happens in the C library code without returning to the interpreter after each iterator element.

score 0 · Answer 2 · answered Oct 19 '11 at 02:50

0

There's no "trick" per se; running compiled C code is faster than running Python bytecode.

answered Oct 19 '11 at 02:50

Ignacio Vazquez-Abrams

776,304
153
1,341
1,358

what's the trick behind python's any() function?

2 Answers2