56

Prompted by the discussion here

The docs suggest some equivalent code for the behaviour of all and any

Should the behaviour of the equivalent code be considered part of the definition, or can an implementation implement them in a non-shortcircuit manner?

Here is the relevant excerpt from cpython/Lib/test/test_builtin.py

def test_all(self):
    self.assertEqual(all([2, 4, 6]), True)
    self.assertEqual(all([2, None, 6]), False)
    self.assertRaises(RuntimeError, all, [2, TestFailingBool(), 6])
    self.assertRaises(RuntimeError, all, TestFailingIter())
    self.assertRaises(TypeError, all, 10)               # Non-iterable
    self.assertRaises(TypeError, all)                   # No args
    self.assertRaises(TypeError, all, [2, 4, 6], [])    # Too many args
    self.assertEqual(all([]), True)                     # Empty iterator
    S = [50, 60]
    self.assertEqual(all(x > 42 for x in S), True)
    S = [50, 40, 60]
    self.assertEqual(all(x > 42 for x in S), False)

def test_any(self):
    self.assertEqual(any([None, None, None]), False)
    self.assertEqual(any([None, 4, None]), True)
    self.assertRaises(RuntimeError, any, [None, TestFailingBool(), 6])
    self.assertRaises(RuntimeError, all, TestFailingIter())
    self.assertRaises(TypeError, any, 10)               # Non-iterable
    self.assertRaises(TypeError, any)                   # No args
    self.assertRaises(TypeError, any, [2, 4, 6], [])    # Too many args
    self.assertEqual(any([]), False)                    # Empty iterator
    S = [40, 60, 30]
    self.assertEqual(any(x > 42 for x in S), True)
    S = [10, 20, 30]
    self.assertEqual(any(x > 42 for x in S), False)

It doesn't do anything to enforce the shortcircuit behaviour

smci
  • 32,567
  • 20
  • 113
  • 146
John La Rooy
  • 295,403
  • 53
  • 369
  • 502
  • Interesting that the test-suite doesn't enforce short-circuiting. Seems like an oversight to me. I still maintain that short-circuiting is part of the specification though. – mgilson Feb 06 '13 at 13:56
  • 5
    I spotted a bug in the code you posted and filed [an issue](http://bugs.python.org/issue17142). – Sjoerd Feb 06 '13 at 15:13
  • 6
    I also filed [an issue](http://bugs.python.org/issue17255), for the behaviour in question. – wim Mar 12 '13 at 04:54
  • 1
    @wim, Nice to be able to finally settle the issue :) – John La Rooy Mar 12 '13 at 04:58

4 Answers4

75

The behaviour is guaranteed. I've contributed a patch, which was accepted and merged recently, so if you grab the latest sources you will see that the short-circuiting behaviour is now explicitly enforced. 

git clone https://github.com/python/cpython.git
grep Short-circuit cpython/Lib/test/test_builtin.py
wim
  • 338,267
  • 99
  • 616
  • 750
  • 33
    It's wonderful to see the timeline on this question: Feb 13 the question is asked, Feb 14 there's the [first revision](http://stackoverflow.com/revisions/14866380/1) of this answer, Feb 20 the [bug/patch](http://bugs.python.org/issue17255) is created, Feb 21 it's merged, and Feb 23 this answer is updated. Thanks to this question being asked on Stack Overflow, the Python library has become better. (Thanks to you @wim of course.) – ShreevatsaR Mar 18 '16 at 00:57
17

The docs say

"Return True if any element of the iterable is true. If the iterable is empty, return False. EQUIVALENT TO:" (emphasis mine) ...

def any(iterable):
    for element in iterable:
        if element:
            return True
    return False

If any didn't short circuit, it wouldn't be EQUIVALENT to the posted code since the posted code clearly short circuits. You could consume more of a generator than you want to for example. In light of that, I say that the short circuiting behavior is guaranteed.

The exact same argument could be made for all.

mgilson
  • 300,191
  • 65
  • 633
  • 696
  • 2
    @gnibbler -- I don't think that there will be any counter-arguments (other than perhaps by Simon on your original post). As far as I'm concerned, this is clearly documented behavior ... – mgilson Feb 06 '13 at 13:58
  • @gnibbler -- If you're looking for interesting discussions on the implementation, see the comments on `exec` vs. `locals` between myself, MartijnPieters and DSM that started [here](http://stackoverflow.com/a/14697234/748858) and ended up [here](http://stackoverflow.com/a/14716991/748858) – mgilson Feb 06 '13 at 14:01
  • 1
    I don't think there *are* any counter-arguments; both `any` and `all` short-circuit according to the spec for the reasons given. One can raise purely stylistic questions about using this for iterator-advancement, but it'll always do the right thing. – DSM Feb 06 '13 at 14:02
13

Note that this does not answer the OP's very different question

In case you landed here wondering why any/all calls can seem not to short circuit

One reason is a false expectation: using a list comprehension inside the call and expecting the building of the list to short-circuit:

>>> def print_and_return_num(num):
    print_and_return_num(num)
    return num
...

>>> any(print_and_return_num(num) for num in [1, 2, 3, 4])
1
True

>>> any([print_and_return_num(num) for num in [1, 2, 3, 4]])
1
2
3
4
True

In the second example, the list comprehension gets evaluated first: the entire list gets built before any() can look at it. This is the expected behavior, but it may take a second to see it etc.

scharfmn
  • 3,561
  • 7
  • 38
  • 53
  • Downvoter: I have clarified the example and the fact that I am not answering the question. – scharfmn May 07 '19 at 16:05
  • 2
    Good pitfall to note. In the simpler case `any((a, b, c, d))` all `a`,`b`,`c`,`d` will be evaluated _before_ the any starts. – russianfool Jun 11 '19 at 01:46
  • Also worth noting: `any(hi(), hi(), hi(), hi())` throws `TypeError: any() takes exactly one argument (4 given)` – scharfmn Jul 10 '20 at 05:08
  • 1
    Right, the any() function takes an iterable. This means you can use it with a generator expression without adding an "extra layer" of [] or () (creating a list and tuple respectively), but must wrap individual members if they're not an iterator. Not super applicable here, but it's good to watch out for members that are themselves iterators (e.g. a string), as it'll run but incorrectly. – russianfool Jul 10 '20 at 22:44
  • 2
    This answer (with some elaboration) would be better placed at https://stackoverflow.com/questions/62987814. – Karl Knechtel Aug 02 '22 at 22:16
  • really want out with pylab/importing `any` from numpy, it has different behavoir – PascalVKooten Sep 24 '22 at 09:41
3

It HAS to short circuit, since it could be given an unbound iterable. If it did not short circuit then this would never terminate:

any(x == 10 for x in itertools.count())
Dave Kirby
  • 25,806
  • 5
  • 67
  • 84
  • 2
    I don't think that this example proves that it *has* to short-circuit. You can write statements like this which never terminate quite easily and python gives you the freedom to do that: `any( x == -1000 for x in itertools.count() )` – mgilson Feb 06 '13 at 13:42
  • 1
    @mgilson But not short-circuiting changes a program (fragment) from terminating to not terminating. Python also gives you the power to print output, but that doesn't mean integer addition may print its result. –  Feb 06 '13 at 13:47
  • 3
    Not terminating in this case would suck - and make any/all much less useful. My question is whether or not the Python language _requires_ it to terminate. – John La Rooy Feb 06 '13 at 13:47
  • 5
    @delnan -- The question is about whether python is *required* to terminate in this case. I'm saying that arguing that you can write a case where `any` does not terminate in the case of no short-circuiting is not sufficient to prove that the behavior is guaranteed. You can only get that information from the documentation. If python exhibits different behavior than the documentation, then it's a bug in either the docs or the implementation (usually the latter). – mgilson Feb 06 '13 at 13:53