Lazy store and code execution in Python functions with optional return values

Question

If I have a function that returns multiple outputs, from which some aren't needed at times, I found that it is a convention to "ignore" these outputs with underscore, e.g.

a, _ = fn_with_two_outputs()

Until now I assumed that the value is just dropped and naively hoped that the compiler is clever enough not to execute any code associated with the value. But in a recent debugging session I noticed that this method actually creates a variable _ with the output - which is a waste of memory and should discourage from using the underscore-convention, shouldn't it?

I'm now leaning to this variant:

a = fn_with_two_outputs()[0]

that I found among other ways of "ignoring output", since as far as I tested it, at least it doesn't store the output fn_with_two_outputs()[1] as a variable in the environment calling the function (it IS stored in the function environment itself). To avoid discussion: To get several return values without calling twice one can use slicing, e.g. fn_with_two_outputs()[0:4].

But: The return value - if "ignored" or not - always seems to be evaluated. I tested this for three variants of "ignoring" it with below code.

def fn_with_two_outputs():
    a = [1, 2, 3]
    b = [0, 0, 0]
    return a, needless_fn(b)

def needless_fn(b):
    print("\"Be so good they can't ignore you.\" - Steve Martin")
    return b

# I want to ignore output b (wrapped by needless_fn)
a, _  = fn_with_two_outputs()   # stores _ = b, needless_fn was executed
a, *b = fn_with_two_outputs()   # suggested for Python3, stores a list b = [b], needless_fn was executed
a = fn_with_two_outputs()[0]    # b nowhere to be found, but needless_fn was executed

Is there a convenient way to actually ignore the output IN the function itself, causing the optional output not to consume any memory and cause code execution? My first idea is, give fn() a parameter to branch between two return statements, one "lazy", the other not - but that seems more like a workaround than an elegant solution.

Unless you pass some custom flag to the function or convert it to the generator, no. — vaultah, Apr 06 '18 at 13:44
How much memory is your unneeded second return value occupying? Is it actually a concern? — Ned Batchelder, Apr 06 '18 at 16:17

score 4 · Answer 1 · edited Apr 06 '18 at 14:08

The function should be changed. Either it does two independent things or the second thing depends on the first. If they are independent, then these things should be two functions.

If they depend on each other, make it two functions, like this:

a = step_a()
b = step_b(a)

If you only want a, then you just write the first line. If you want the expensive computation to yield b, then call the second function. This way the user only has to pay for the things that they want to have computed.

In case the dependencies are the other way around you will always have to do the expensive computation anyway.

score 1 · Accepted Answer · answered Apr 06 '18 at 13:52

There is no (easy) way to affect the called function from the caller without informing the said function of the desired behavior - simply put, your return a, needles_fn(b) will execute needless_fn(b) before it even gets the chance to return the result to the caller.

What you can do is create is a partial function call and let the external caller do the call if it needs the result, something like:

import functools

def fn_with_two_outputs():
    a = [1, 2, 3]
    b = [0, 0, 0]
    return a, functools.partial(needless_fn, b)

def needless_fn(b):
    print("\"Be so good they can't ignore you.\" - Steve Martin")
    return b

print(fn_with_two_outputs()[0])  # needless_fn(b) doesn't get called, [0, 0, 0] is dropped
# [1, 2, 3] 

a, b = fn_with_two_outputs()  # needless_fn(b) doesn't get called, [0, 0, 0] is in memory
print(a)
# [1, 2, 3]
print(b())  # needless_fn(b) is finally evaluated
# "Be so good they can't ignore you." - Steve Martin
# [0, 0, 0]

Beware that in the second case, as stated, [0, 0, 0] still remains in memory as long as there is a reference to b.

Sadly, unless you're talking about instance properties where you can use the magic __getattr__(), you need to explicitly call b() instead of just b for evaluation.

score 0 · Answer 3 · answered Apr 06 '18 at 14:03

Obviously you can't ignore part of a function that you call, because you are a consumer. Every instruction needs to be evaluated. Nevertheless, you can get rid of data returned, by garbage collect manually referred variable.

Even in pure functional programming languages like Haskell, you are only sure the function doesn't alter memory only with a return value, in this case too, it does change memory.

The only way you can deal with a function, is through input data, so you can't influence it's execution normally.

Back to how Python garbage collector deals with it, as far as you call a function, it has reference in memory; to know more check this

Lazy store and code execution in Python functions with optional return values

3 Answers3