Why do list/dict attributes retain values outside decorators, but integer attributes do not?

Question

In the code below, why does g.__int equal 0 instead of 1 at the end, when both g.__dictionary and g.__list retain the values they had inside the decorators?

Why can I add a list/dict as an attribute to a function inside a decorator then access it outside the decorator, but I can't do the same with an integer?

Here's some code to illustrate:

import functools

def dictify(func):
    func.__dictionary = { 0 : 0 }

    @functools.wraps(func)
    def _func(*args, **kwds):
        func.__dictionary[0] += 1
        print('  Incremented __dictionary, now __dictionary = {0}'.format(str(func.__dictionary)))
        return func(*args, **kwds)
    return _func

def listify(func):
    func.__list = [1, 2, 3]

    @functools.wraps(func)
    def _func(*args, **kwds):
        func.__list.append(0)
        print('  Appended 0 to __list, now __list = {0}'.format(str(func.__list)))
        return func(*args, **kwds)
    return _func

def intify(func):
    func.__int = 0

    @functools.wraps(func)
    def _func(*args, **kwds):
        func.__int += 1
        print('  Incremented __int, now __int = {0}'.format(func.__int))
        return func(*args, **kwds)
    return _func

def g():
    return 'pennyroyal tea'

print('*** UNMODIFIED ***')
print('g() returns \'{0}\''.format(g()))
print('id(g) = {0}'.format(id(g)))

g = dictify(g)
print('*** DICTIFIED ***')
print('g() returns \'{0}\''.format(g()))
print('g.__dictionary = {0}'.format(str(g.__dictionary)))
print('id(g) = {0}'.format(id(g)))

g = listify(g)
print('*** LISTIFIED ***')
print('g() returns \'{0}\''.format(g()))
print('g.__dictionary = {0}'.format(str(g.__dictionary)))
print('g.__list = {0}'.format(str(g.__list)))
print('id(g) = {0}'.format(id(g)))

g = intify(g)
print('*** INTIFIED ***')
print('g() returns \'{0}\''.format(g()))
print('g.__dictionary = {0}'.format(str(g.__dictionary)))
print('g.__list = {0}'.format(str(g.__list)))
print('g.__int = {0}'.format(str(g.__int)))
print('id(g) = {0}'.format(id(g)))

This gives the following output:

*** UNMODIFIED ***
g() returns 'pennyroyal tea'
id(g) = 139861398390976
*** DICTIFIED ***
  Incremented __dictionary, now __dictionary = {0: 1}
g() returns 'pennyroyal tea'
g.__dictionary = {0: 1}
id(g) = 139861398391096
*** LISTIFIED ***
  Appended 0 to __list, now __list = [1, 2, 3, 0]
  Incremented __dictionary, now __dictionary = {0: 2}
g() returns 'pennyroyal tea'
g.__dictionary = {0: 2}
g.__list = [1, 2, 3, 0]
id(g) = 139861398391216
*** INTIFIED ***
  Incremented __int, now __int = 1
  Appended 0 to __list, now __list = [1, 2, 3, 0, 0]
  Incremented __dictionary, now __dictionary = {0: 3}
g() returns 'pennyroyal tea'
g.__dictionary = {0: 3}
g.__list = [1, 2, 3, 0, 0]
g.__int = 0
id(g) = 139861398391336

You can see that inside the decorator, the value of func.__int is printed as 1, but outside the decorator, g.__int is the default 0, yet g.__dictionary and g.__list retain their values when referenced both inside and outside their decorators.

Note: The id(g) calls show that decorating with dictify(), listify(), and intify() all return new objects, illustrating that functions are immutable. (See my detailed explanation here)

This question is based on a previous one of mine here. My practical need was satisfied by its answers, but my 'why' instinct won't leave this alone. :)

Answer 1

You assign the attributes on the function passed to the decorator, but you return a different function (the wrapped function). functools.wraps shallow-copies attributes from one to the other, which means it copies the list and dict objects. You then mutate these objects. But you can't mutate the int, so all you are doing is changing the value on the "unwrapped" version of g, while you print the value of the wrapped version.

Here is an illustrative thing to try:

>>> def g():
...     return 'pennyroyal tea'
>>> f = intify(g)
>>> f()
  Incremented __int, now __int = 1
'pennyroyal tea'
>>> f.__int
0
>>> g.__int
1

I intified g but assigned it to f. You can see that the __int attribute is updated --- but on the original function, not the wrapped one.

You don't see the difference with the list and dict, because these objects are mutable. There is one list and one dict, both shared by both functions. However, you can again see it if you split out the wrapped functions one by one:

>>> f = dictify(g)
... f2 = listify(f)
... f3 = intify(f2)
>>> f3()
  Incremented __int, now __int = 1
  Appended 0 to __list, now __list = [1, 2, 3, 0]
  Incremented __dictionary, now __dictionary = {0: 1}
'pennyroyal tea'
>>> f3.__list is f2.__list
True
>>> f3.__dictionary is f2.__dictionary
True
>>> f3.__int is f2.__int
False

Your modifications to __list and __dictionary mutate the objects, but your modification to __int creates a new int (since ints can't be mutated), creating a separation between the __int attributes of the function you pass into the decorator and the wrapped function it gives back.

The basic problem here is that you what you seem to want to do in the decorator is thisFuncion.__list.append(0), where thisFunction is the returned, decorated function, not the to-be-decorated function. That is, you want the wrapper to be able to refer to itself. But you can't do this. There is no general way in Python for a function to refer to itself. In your decorator, you defined a function _func that refers to a function func. There are two different functions, _func is only setting attributes on func, not on itself.

Of course, the real question is why you'd try to set function attributes like this in the first place. But I gather from your question that you're asking just out of curiosity to understand what's going on, not because you actually want to do this.

Answer 2

Adding to @BrenBarn's answer, If you write your own version of functools.wraps then you'll have to do something like this:

def intify(func):
    print func is g
    func.__int = 0
    def _func(*args, **kwds):
        func.__int += 1
        print('  Incremented __int, now __int = {0}'.format(func.__int))
        return func(*args, **kwds)
    _func.__int = func.__int
    _func.__doc__ = func.__doc__
    #... and some more 
    return _func

i.e You'll have to assign the integer value to the new function object:

_func.__int = func.__int

But, as integers are immutable so changing one reference doesn't affect the other:

>>> x = 1
>>> y = x 
>>> x += 1
>>> x       
2
>>> y         #y still unchanged
1

But same thing is not applicable for mutable objects when you perform in-place operations:

>>> x = [1]
>>> y = x
>>> y.append(10)
>>> x
[1, 10]
>>> y
[1, 10]
>>> 

From the source code you can see that wraps simply assigns the same objects to the new function object, hence you get that side effect when you use mutable objects.

"""Update a wrapper function to look like the wrapped function

   wrapper is the function to be updated
   wrapped is the original function
   assigned is a tuple naming the attributes assigned directly
   from the wrapped function to the wrapper function (defaults to
   functools.WRAPPER_ASSIGNMENTS)
   updated is a tuple naming the attributes of the wrapper that
   are updated with the corresponding attribute from the wrapped
   function (defaults to functools.WRAPPER_UPDATES)
"""
for attr in assigned:
    setattr(wrapper, attr, getattr(wrapped, attr))
for attr in updated:
    getattr(wrapper, attr).update(getattr(wrapped, attr, {}))
# Return the wrapper so this can be used as a decorator via partial()
return wrapper