Deleting variables after use during assignment

Question

I tend to see code where it is not clear where you are in the computation when looking at an individual line. For example:

def forward(self, X):
    X = layer(X)
    X = activation(X)
    X = layer(X)
    X = activation(X)
    return X

Clearly, lines 2 and 4 as well as 3 and 5, are indistinguishable from eachother.

I would much rather write more readable code, where it is clear where you are in the computation when looking at a particular line of code. For example (using Andrew Ng's notation):

def forward(self, X):
    A0 = X
    Z1 = layer(A0)
    A1 = activation(Z1)
    Z2 = layer(A1)
    A2 = activation(Z2)
    return A2

But this could cause memory errors given large variables, so it may be necessary to do the following...

def forward(self, X):
    A0 = X
    Z1 = layer(A0); del A0
    A1 = activation(Z1); del Z1
    Z2 = layer(A1); del A1
    A2 = activation(Z2); del Z2
    return A2

However, this compromizes the readability does not take advantage of garbage collector properties. I'm not sure if this alternative makes a difference.

If A0, Z1, A1, Z2, A2 were all the same size, there would ideally be a solution that only took up the size of at most two of the variables in memory.

Is there a pythonic way to delete all the variables used on the right-hand side after calculating the left-hand side, or have some similar effect? Alternatively, could you create a function that does the deletion and assignment (given scoping limitations) and what's the efficiency tradeoff there?

This might help. https://stackoverflow.com/questions/26545051/is-there-a-way-to-delete-created-variables-functions-etc-from-the-memory-of-th?rq=1 — bhatnaushad, Feb 28 '19 at 07:09
Assuming the content of `A0` is still referenced by whatever the function inside `layer()` did, the only extra cost of the nice variable names is a local reference to to object. Stick with the readable code, and only worry about the extra complexity if you actually have a problem. — solidpixel, Feb 28 '19 at 07:52

gilch · Accepted Answer · 2019-03-01T03:26:23.413

there would ideally be a solution that only took up the size of at most two of the variables in memory.

del is not often needed in Python. The repetition is a code smell. Don't Repeat Yourself (DRY principle). You can remove the repetition by using a loop.

def forward(self, A):
    for _ in range(2):
        Z = layer(A)
        A = activation(Z)
    return A

You'd be re-using the two variables A and Z.

You can compress this further by nesting the calls, which removes the Z altogether.

def forward(self, A):
    for _ in range(2):
        A = activation(layer(A))
    return A

If you're functionally inclined, this pattern is known as a "reduce" (sometimes also called a "fold"). This may be less "Pythonic", but the functional style is still pretty commonly used in Python code.

from functools import reduce

def forward(self, X):
    return reduce(lambda A, _: activation(layer(A)), range(2), X)

Or even,

def forward(self, X):
    return reduce(lambda x, f: f(x), [layer, activation]*2, X)

The popular toolz library also implements this pattern

from toolz.functoolz import thread_first

def forward(self, X):
    return thread_first(X, layer, activation, layer, activation)

No intermediate variables required, but you can add comments if it makes you feel better.

def forward(self, X):
    return thread_first(
        X,  # A0
        layer, # Z1
        activation, # A1
        layer,  # Z2
        activation,  # A2
    )

These aren't enforced or anything.

In fact, the variables are not required at all, save for the parameter.

def forward(self, X):
    return activate(layer(activate(layer(X))))

The function really is that simple, and the fuss about variable names seems to be over-complicating it.

For just two layers, this is probably OK, but the looping/reducing version makes it easier to add more layers later by updating the range() argument, which could even be another parameter to the .forward() method.

could you create a function that does the deletion and assignment (given scoping limitations) and what's the efficiency tradeoff there?

You can't really delete locals except with del (or when they go out of scope). But instead of locals, you could make your own namespace. This is backed by a dict, which is only slightly less efficient than locals, not enough to matter here.

from types import SimpleNamespace

class MonoNamespace(SimpleNamespace):
    """A namespace that holds only one attribute at a time."""
    def __setattr__(self, name, value):
        vars(self).clear()
        vars(self)[name] = value

def forward(self, X):
    ns = MonoNamespace(A0=X)
    ns.Z1 = layer(ns.A0)
    ns.A1 = activation(ns.Z1)
    ns.Z2 = layer(ns.A1)
    ns.A2 = activation(ns.Z2)
    return ns.A2

The last example is a complete antipattern - properties should not do costly computation -, and most of the previous ones don't help in making the code easier to read, trace and maintain. Actually, the first snippet is certainly the most readable and maintainable (hence the most pythonic). — bruno desthuilliers, Feb 28 '19 at 08:44
@brunodesthuilliers it's only syntax, but I don't exactly disagree in the normal case. I was talking about what is possible, not what is conventional. That `@property` example wasn't very relevant and I have removed it. — gilch, Mar 01 '19 at 03:23

Deleting variables after use during assignment

1 Answers1