Pythonic way to cycle through purely side-effect-based comprehension

Question

What is the most pythonic way to execute a full generator comprehension where you don't care about the return values and instead the operations are purely side-effect-based?

An example would be splitting a list based on a predicate value as discussed here. It's natural to think of writing a generator comprehension

split_me = [0, 1, 2, None, 3, '']
a, b = [], []
gen_comp = (a.append(v) if v else b.append(v) for v in split_me)

In this case the best solution I can come up with is to use any

any(gen_comp)

However that's not immediately obvious what's happening for someone who hasn't seen this pattern. Is there a better way to cycle through that full comprehension without holding all the return values in memory?

Answer 1

You do so by not using a generator expression.

Just write a proper loop:

for v in split_me:
    if v:
        a.append(v)
    else:
        b.append(v)

or perhaps:

for v in split_me:
    target = a if v else b
    target.append(v)

Using a generator expression here is pointless if you are going to execute the generator immediately anyway. Why produce an object plus a sequence of None return values when all you wanted was to append values to two other lists?

Using an explicit loop is both more comprehensible for future maintainers of the code (including you) and more efficient.

Answer 2

itertools has this consume recipe

def consume(iterator, n):
    "Advance the iterator n-steps ahead. If n is none, consume entirely."
    # Use functions that consume iterators at C speed.
    if n is None:
        # feed the entire iterator into a zero-length deque
        collections.deque(iterator, maxlen=0)
    else:
        # advance to the empty slice starting at position n
        next(islice(iterator, n, n), None)

in your case n is None, so:

collections.deque(iterator, maxlen=0)

Which is interesting, but also a lot of machinery for a simple task

Most people would just use a for loop

Answer 3

There's nothing non-pythonic in writing things on many lines and make use of if-statements:

for v in split_me:
    if v:
        a.append(v)
    else:
        b.append(v)

If you want a one-liner you could do so by putting the loop on one line anyway:

for v in split_me: a.append(v) if v else b.append(v)

If you want it in an expression (which still beats me why you want unless you have a value you want to get out of it) you could use list comprehension to force looping:

[x for x in (a.append(v) if v else b.append(v) for v in split_me) if False]

Which solution do you think best shows what you're doing? I'd say the first solution. To be pythonic you should probably consider the zen of python, especially:

• Readability counts.
• If the implementation is hard to explain, it's a bad idea.

Answer 4

As others have said, don't use comprehensions just for side-effects.

Here's a nice way to do what you're actually trying to do using the partition() recipe from itertools:

try:  # Python 3
    from itertools import filterfalse
except ImportError:  # Python 2
    from itertools import ifilterfalse as filterfalse
    from itertools import ifilter as filter


from itertools import tee


def partition(pred, iterable):
    'Use a predicate to partition entries into false entries and true entries'
    # From itertools recipes:
    # https://docs.python.org/3/library/itertools.html#itertools-recipes
    # partition(is_odd, range(10)) --> 0 2 4 6 8   and  1 3 5 7 9
    t1, t2 = tee(iterable)
    return filterfalse(pred, t1), filter(pred, t2)

split_me = [0, 1, 2, None, 3, '']

trueish, falseish = partition(lambda x: x, split_me)

# You can iterate directly over trueish and falseish,
# or you can put them into lists

trueish_list = list(trueish)
falseish_list = list(falseish)

print(trueish_list)
print(falseish_list)

Output:

[0, None, '']
[1, 2, 3]

Answer 5

Just to throw in another reason why using any() to consume a generator is a horrible idea, you need to remember that any() and all() are guaranteed to do short-circuit evaluation which means that if the generator ever returns a True value then all() will early-out on you and leave your generator incompletely consumed.

This is adding an extra conditional test / stop condition that you A) probably don't want, and B) may be far away from where the generator is created.

Many standard library functions return None so you could get away with all() for a while until suddenly it's not doing what you expect, and you might stare at that code for a long time before it occurs to you if you've gotten into the habit of using all() in this way.

If you must do something like this, then itertools.consume() is really the only reasonable way to do it I think.

Answer 6

any is short, but is not a general solution. Something which works for any generator is the straightforward

for _ in gen_comp: pass

which is also shorter and more efficient than a generally working any method,

any(None for _ in gen_comp)

so the for loop is really the clearest and best. Its only downside is that it cannot be used in expressions.