How to make super() work in this non-ideal situation in python?

Question

class ClsOne(object):
    def __init__(self):
        super(ClsOne, self).__init__()
        print "Here's One"

class ClsTwo(ClsOne):
    def __init__(self):
        super(ClsTwo, self).__init__()
        print "Here's Two"

class ClsThree(ClsTwo): # Refer to one blackbox object
    def __init__(self):
        # super(ClsThree, self).__init__()
        print "Here's Three"

class ClsThreee(ClsTwo): # Refer to your custom object
    def __init__(self):
        super(ClsThreee, self).__init__()
        print "Here's Threee"

class ClsFour(ClsThree, ClsThreee): # Multiple Inheritance
    def __init__(self):
        super(ClsFour, self).__init__()
        print "Here's Four"

entity = ClsFour()

In this case, you are trying to combine ClsThree (which comes from a single compiled library and is very hard to change) and your own ClsThreee object together. Because ClsThree forgets to call super() in its constructor, their kid cannot execute ClsThreee's constructor when it uses super().

As a result, the output will just like this:

Here's Three
Here's Four

Obviously, I can manually call every bases of ClsFour rather than use super(), but it's a bit complicated when this problem scattered all over my codebase.

By the way, that blackbox stuff is PySide :)

SUPPLEMENT:

Thanks to @WillemVanOnsem and @RaymondHettinger, the previous ClsFour question is solved. But with some further investigations, I found the similar problem in PySide doesn't have same concept.

In ClsFour context, if you try to run:

print super(ClsFour, self).__init__

You'll get:

<bound method ClsFour.__init__ of <__main__.ClsFour object at 0x00000000031EC160>>

But in the following PySide context:

import sys
from PySide import QtGui

class MyObject(object):
    def __init__(self):
        super(MyObject, self).__init__()
        print "Here's MyObject"

class MyWidget(QtGui.QWidget, MyObject):
    def __init__(self):
        super(MyWidget, self).__init__()

app = QtGui.QApplication(sys.argv)
widget = MyWidget()
print super(MyWidget, widget).__init__

The result is:

<method-wrapper '__init__' of MyWidget object at 0x0000000005191D88>

It doesn't print "Here's MyObject" and the init attribute of super() has a different type as well. Previously, I try to simplify this problem as the ClsFour. But now I think it isn't totally the same.

I guess the problem occurs in shiboken library but I'm not sure.

TIPS:

The problem also appear in PyQt context, but you can make MyObject inherit from QObject to solve. This solution is useless in PySide.

Don't know what the answer is but I have to say, I'm loving your naming scheme. — stelioslogothetis, Jul 28 '17 at 16:24
Yes it would defenitely help if your would call your classes for what the are... e.g. call ClsThree ClassFromLibraryCannotBeChanged and so on... — Jan, Jul 28 '17 at 16:28
Not just ClsThree. Cls**Two** isn't performing superclass initialization either. — user2357112, Jul 28 '17 at 17:53
Possible duplicate of https://stackoverflow.com/questions/1848474/method-resolution-order-mro-in-new-style-classes — NoDataDumpNoContribution, Aug 08 '17 at 11:36

score 3 · Accepted Answer · edited Jul 29 '17 at 14:47

super() is a proxy object that uses the Method Resolution Order (MRO) to determine what method to call when you perform a call on super().

If we inspect the __mro__ of ClassFour, we get:

>>> ClsFour.__mro__
(<class '__main__.ClsFour'>, <class '__main__.ClsThree'>, <class '__main__.ClsThreee'>, <class '__main__.ClsTwo'>, <class '__main__.ClsOne'>, <type 'object'>)

Or made it shorter myself (not the Python output):

>>> ClsFour.__mro__
(ClsFour, ClsThree, ClsThreee, ClsTwo, ClsOne, object)

Now super(T,self) is a proxy object that uses the MRO from (but excluding) T. So that means that super(ClsFour,self) is a proxy object that works with:

(ClsThree, ClsThreee, ClsTwo, ClsOne, object)  # super(ClsFour,self)

What will happen if you query an attribute (a method is also an attribute) of a class is that Python will walk through the MRO and inspect whether the element has such attribute. So it will first inspect whether ClsThree has an __init__ attribute, if not it will continue to look for it in ClsThreee and so on. From the moment it finds such attribute it will stop, and return it.

So super(ClsFour,self).__init__ will return the ClsThree.__init__ method. The MRO is also used to find methods, attributes, etc. that are not defined on the class level. So if you use self.x and x is not an attribute of the object nor of the ClsFour object, it will again walk through the MRO in search for x.

If you want to call the __init__ of all the direct parents of ClassFour you can use:

class ClsFour(ClsThree, ClsThreee):
    def __init__(self):
        # call *all* *direct* parents __init__
        for par in ClsFour.__bases__:
            par.__init__(self)

Which is probably the most elegant, since if the bases change, it will still work. Note that you have to make sure that __init__ exists for every parent. Since it is however defined at the object level, we can safely assume this. For other attributes however, we can not make that assumption.

EDIT: Mind that as a result super() does not necessary points to the parents, grandparents and/or ancestors of that class. But to parent classes of the object.

The super(ClsThree,self) in the ClsThree class, will - given it is an ClsFour object, work with the same mro (since it takes the mro from the self). So super(ClsThree,self) will inspect the following sequence of classes:

(ClsThreee, ClsTwo, ClsOne, object)

For instance, if we write (out of the scope of any class) super(ClsTwo,entity).__init__(), we get:

>>> super(ClsTwo,entity).__init__()
Here's One
>>> super(ClsThree,entity).__init__()
Here's Two
Here's Threee
>>> super(ClsThreee,entity).__init__()
Here's Two
>>> super(ClsFour,entity).__init__()
Here's Three

If super() will just return the first method in MRO. Why it can execute the second base class if the first one doesn't lose its super(). — Calvin, Jul 28 '17 at 16:45
What do you mean? Of course `ClsThree` can call `super(..)` as well and so you can create a chain of super calls. — Willem Van Onsem, Jul 28 '17 at 16:47
Ah, I mean that ClsThreee isn't the base of ClsThree. But it's still executed when ClsFour perform a call on super() if ClsThree doesn't lose its super(). — Calvin, Jul 28 '17 at 16:58
@Calven: mind that the super of `ClsThree` has the *same mro*, and thus only starts later. So the `super(ClsThree)` works with `(ClsThreee,ClsTwo,ClsOne,object)`. — Willem Van Onsem, Jul 28 '17 at 17:42
Calling the `__init__` methods of all direct superclasses directly will lead to the same `__init__` methods getting called repeatedly in common cases; it only doesn't here because ClsThree (and ClsTwo) forget superclass initialization entirely instead of just not using `super`. Also, you probably want `ClsFour.__bases__` instead of `self.__class__.__bases__`, for pretty much the same reasons `super(self.__class__, self)` is wrong, and you're missing the `self` argument in `par.__init__`. — user2357112, Jul 28 '17 at 17:57
@user2357112: well `super(self.__class__,self)` evidently will - regardless of the class where it is called - lead to the same method - and thus can cause an infinite loop :). I do not advocate calling all the parents directly as a good remedy. It was meant to show that the two are different :). — Willem Van Onsem, Jul 28 '17 at 18:00
If someone makes a subclass of `ClsFour` with your code, the loop over `self.__class__.__bases__` will call `ClsFour.__init__` over and over in an infinite loop. — user2357112, Jul 28 '17 at 18:14
@user2357112: I have found this issue of using loop on bases. That's definitely not the right way for solving this question. — Calvin, Jul 29 '17 at 04:49
@WillemVanOnsem: Brilliant supplement. It makes scene now. Thanks a lot. — Calvin, Jul 29 '17 at 04:53
@WillemVanOnsem: But with further investigation, I found this PySide problem is not as easy as I imagined, so I post something new on my question. — Calvin, Jul 29 '17 at 05:00

Raymond Hettinger · Answer 2 · 2017-07-28T18:36:08.837

Summary

Because ClsThree forgets to call super() in its constructor, their kid cannot execute ClsThreee's constructor when it uses super().

This is covered in the "How to Incorporate a Non-cooperative Class" section of Super Considered Super blog post.

The key is to create an adapter class that plays by the rules to act cooperatively by calling super(). Use that adapter to wrap the original class.

Worked-out code

In the code below, AdaptThree is the new adapter class and ClsFour now inherits from AdaptThree instead of the original blackbox non-cooperative class.

class ClsOne(object):
    def __init__(self):
        print "Here's One"

class ClsTwo(ClsOne):
    def __init__(self):
        print "Here's Two"

class ClsThree(ClsTwo): # Refer to one blackbox object
    def __init__(self):
        # super(ClsThree, self).__init__()
        print "Here's Three"

class AdaptThree(object):
    def __init__(self):
        _three = ClsThree()
        super(AdaptThree, self).__init__()          

class ClsThreee(ClsTwo): # Refer to your custom object
    def __init__(self):
        super(ClsThreee, self).__init__()
        print "Here's Threee"

class ClsFour(AdaptThree, ClsThreee): # Multiple Inheritance
    def __init__(self):
        super(ClsFour, self).__init__()
        print "Here's Four"

entity = ClsFour()

This outputs:

Here's Three
Here's Two
Here's Threee
Here's Four

Other details

In OP's example, ClsTwo also looks to be non-cooperative and should wrapped as well.
Presumably these classes with have methods other than initialization. The adapter classes will need to wrap and dispatch those calls as well.
Depending on the application, it may be easier to use composition rather than inheritance.

Thanks for your answer. I've read this popular blog before. I didn't choose this way due to your second detail. There're lots of other methods need to adapt as well and that's quite a heavy work. By the way, I've updated my question, I will appreciate your ideas for my supplement. — Calvin, Jul 29 '17 at 05:49
@Calven Have you tried writing a \__getattribute__ to auto-dispatch the remaining methods. — Raymond Hettinger, Jul 29 '17 at 06:37
Do you mean something like self.__dict__.update(adapter.__dict__)? — Calvin, Jul 29 '17 at 09:35

score 0 · Answer 3 · answered Jul 29 '17 at 15:05

Regarding the issue specific to PySide/PyQt4: one solution is to ensure that any mixins always preceed the Qt classes in the base-class definition:

import sys
from PySide import QtGui

class MyObject(object):
    def __init__(self, parent=None, other=None):
        super(MyObject, self).__init__(parent)
        print "Here's MyObject: %r" % other

class MyWidget(MyObject, QtGui.QWidget):
    def __init__(self, parent=None, other=None):
        super(MyWidget, self).__init__(parent, other)

app = QtGui.QApplication(sys.argv)
parent = QtGui.QWidget()
widget = MyWidget(parent, 'FOO')
print super(MyWidget, widget).__init__
# check that the QWidget.__init__ was called correctly
print 'widget.parent() is parent:', widget.parent() is parent

Output:

Here's MyObject: 'FOO'
<bound method MyWidget.__init__ of <__main__.MyWidget object at 0x7f53b92cca28>>
widget.parent() is parent: True

(NB: PyQt5 has improved Support for Cooperative Multi-inheritance, so this issue doesn't arise there).

It works!!! I have tried this solution before but missed the parent parameter at that time. It causes the mixed object cannot be showed. — Calvin, Jul 29 '17 at 16:01

How to make super() work in this non-ideal situation in python?

3 Answers3

Summary

Worked-out code

Other details