3

Is __setattr__ called when an existing attribute is incremented?

I have a class called C, and I'm trying to overload __setattr__. This is a section of code from inside the class C:

class C:

    def bump(self):
        self.a += 1
        self.b += 1
        self.c += 1

    def __setattr__(self,name,value):
        calling = inspect.stack()[1]
        if 'private_' in name:
            raise NameError("\'private_\' is in the variable name.")
        elif '__init__' in calling.function:
            self.__dict__['private_' + name] = value
        elif name.replace('private_', '') in self.__dict__:
            if self.in_C(calling):
                if name.replace('private_', '') in self.__dict__.keys():
                    old_value = self.__dict__[name.replace('private_', '')]
                    new_value = old_value + value
                    self.__dict__[name.replace('private_', '')] = new_value
                else:
                    self.__dict__[name.replace('private_','')] = value
            else:
                raise NameError()
        else:
            self.__dict__[name] = value

__setattr__, according to the Python docs,

object.__setattr__(self, name, value): Called when an attribute assignment is attempted. This is called instead of the normal mechanism (i.e. store the value in the instance dictionary). name is the attribute name, value is the value to be assigned to it.

I know you can assign a value to a variable (ex: C.__dict__[name] = value), but what about when an existing attribute is incremented, like self.a += 1 in bump()?

Assuming that the attributes a, b, and c already already defined, I called bump(), which then called __setattr__. However, I get this error:

Error: o.bump() raised exception TypeError: unsupported operand type(s) for +=: 'NoneType' and 'int'

Is setattr called when an existing attribute is incremented? If so, how would I increment the existing attribute inside setattr?

Note: Assume that bump() is called after a, b, and c are defined. Also, in_C(calling) is a function that checks if __setattr__ was called from __init__, some method inside C, or a method outside of C.

Tell me if any further clarification is needed.

martineau
  • 119,623
  • 25
  • 170
  • 301
Justin Yum
  • 189
  • 2
  • 12

2 Answers2

7

Python: Is __setattr__ called when an existing attribute is incremented?

The answer is yes. This is easily seen with a simplified version of your code:

class C(object):

    def __init__(self, a):
        object.__setattr__(self, 'a', a)

    def __setattr__(self, name, value):
        print('Setting {} to {}'.format(name, value))
        object.__setattr__(self, name, value)


c = C(10)
c.a += 1

Running that snippet produces:

Setting a to 11

The issue with the code you posted is that += calls __getattribute__ first before it calls __setattr__. That is what fails if the attribute doesn't already exist.

The solution is to make sure the attributes are initialized before the call to bump():

class C(object):

    def __init__(self, a, b, c):
        object.__setattr__(self, 'a', a)
        object.__setattr__(self, 'b', a)
        object.__setattr__(self, 'c', a)

In addition to that fix, there are other errors as well (in inspect for example) but this should get you started.

Raymond Hettinger
  • 216,523
  • 63
  • 388
  • 485
  • 1
    Python developer cheat mode! :) – Grimmy Apr 29 '17 at 02:42
  • Why the dirty hack? Isn't `self.a=a` good enough? If that's due to broken logic in `__setattr__`, that logic needs fixing instead rather than piling up more hacks on top of it. – ivan_pozdeev Apr 29 '17 at 02:50
  • @ivan_pozdeev Whew, that seems a little judgmental ;-) The answer is that whenever altering \__setattr__, it changes how the \__init__ method is able to save the attributes in the first place. Depending on what the code is doing in \__setattr__, the \__init__ is broken without the code I posted. Try it and see. – Raymond Hettinger Apr 29 '17 at 02:53
  • I know it's broken. I'm saying this isn't the way to fix it. – ivan_pozdeev Apr 29 '17 at 02:54
  • 1
    @ivan_pozdeev It is a standard technique when programming \__setattr__. There are a number of examples in the standard library. – Raymond Hettinger Apr 29 '17 at 02:58
  • I see only one direct `__setattr__` invocation in the 2.7.13 standard library that is not from an overriding `__setattr__` - in `_threading_local`. And even there, `local.__setattr__` and `local.__getattribute__` breaking Liskov's principle is causing trouble, forcing one to use `object.__getattribute__` and `object.__setattr__` for inherited members everywhere. – ivan_pozdeev Apr 29 '17 at 04:18
  • 2
    @disbeliever I count 13 uses of object.\__setattr__ in the Py3.6 source tree: ``$ ack --type=python 'object.__setattr__'``. Or see this answer by another core developer: http://stackoverflow.com/a/7559315/1001643 . Or just keep downvoting correct answers. – Raymond Hettinger Apr 29 '17 at 04:22
  • I've got the gist of this technique. It divides the `__dict__` into two parts - "normal" attributes accessed with `__getattr__`/`__setattr__` and "special", accessed directly. There are a few problems here: 1) breaks inherited members; 2) breaks users that expect parent's semantics for parent's members; 3) can't add another `__getattr__`/`__setattr__` below without breaking either of them. Having to call `.__whatever__` instead of `self.member` depending on the member is also an unnecessary complication and source of further errors. – ivan_pozdeev Apr 29 '17 at 18:03
  • The right way is to make `__setattr__` transparently treat "normal" and "special" attributes differently. – ivan_pozdeev Apr 29 '17 at 18:13
  • In the 3.6 library, 3 of the uses are not from an overridden `__setattr__` (or a method called exclusively from it, or a different technique entirely), and they all show the above-mentioned problems. (They're in `_threading_local`, `email._policybase` and `multiprocessing.managers`) – ivan_pozdeev Apr 29 '17 at 18:44
  • To drive the point home: the technique _is_ problematic, _is_ used in the Python code base (although much less than a simple grep would suggest), and _does_ cause problems whenever it's used there. – ivan_pozdeev Oct 29 '17 at 08:11
0

Though it may seem that

a += 1

is equivalent to a.__iadd__(1) (if a has __iadd__), it's actually equivalent to:

a = a.__iadd__(1)    # (but `a` is only evaluated once.)

It's just that for mutable types, __iadd__ returns the same object, so you don't see a difference.

Thus, if the target is c.a, c's __setattr__ is called. Likewise, __setitem__ is called if you do something like c['a']+=1.

This is done because Python has immutable types, for which an augmented assignment would otherwise do nothing.

This is documented alright in the grammar reference entry for augmented assignments (emphasis mine):

An augmented assignment evaluates the target (which, unlike normal assignment statements, cannot be an unpacking) and the expression list, performs the binary operation specific to the type of assignment on the two operands, and assigns the result to the original target. The target is only evaluated once.

To illustrate:

In [44]: class C(object):
    ...:     def __init__(self,a):
    ...:         self.a=a
    ...:     def __setattr__(self,attr,value):
    ...:         print "setting `%s' to `%r'"%(attr,value)
    ...:         super(C,self).__setattr__(attr,value)
    ...:     def __setitem__(self,key,value):
    ...:         print "setitem"
    ...:         return setattr(self,key,value)
    ...:     def __getitem__(self,key):
    ...:         return getattr(self,key)
    ...:

In [45]: c=C([])
setting `a' to `[]'

In [46]: c['a']+=[1]
setitem
setting `a' to `[1]'

In [29]: class A(int):
    ...:     def __iadd__(s,v):
    ...:         print "__iadd__"
    ...:         return int.__add__(s,v)
    ...:     def __add__(s,v):
    ...:         print "__add__"
    ...:         return int.__add__(s,v)
    ...:

In [30]: c.a=A(10)
setting `a' to `10'

In [31]: c.a+=1
__iadd__
setting `a' to `11'
ivan_pozdeev
  • 33,874
  • 19
  • 107
  • 152