Is there an implementation of the PyObject class that allows you to properly override magic methods?

Question

So I read in a book, that if you want to extend from built-in types such as list, dict or str, and want to override magic methods, you should use UserList, UserDict and UserString from the collections module respectively instead. Appearently, that is because the base classes are implemented in CPython, where these methods don't call each other, and so, overriding them has no, or unwanted effects.

I was wondering if similar to the UserList class and so on, there exists a class that can be used to 'properly' extend from the object class. I looked at the PyObject documentation and found this, as well as this post, but neither do I want to extend in C, nor do I speak it. I couldn't find anything in the collections module or anywhere else.

The reason why I ask is because of a previous question of mine 'Is it possible to transform default class dunder methods into class methods?' and I have a hunch that the reason why my approach doesn't work is the same one that I outlined earlier.

Can you please clarify what you want to do? The case with ``UserList`` is completely different to the linked question. The methods of builtin objects (such as ``list``) will not call *other* methods of the same object, e.g. ``list.__iter__`` does not use an overwritten ``list.__getitem__`` on a subclass (whether they should do that and in what order is debatable). This does *not* mean you cannot overwrite the special methods. A class' ``__getitem__`` is an entirely separate thing from the class' *class'* (aka metaclass) ``__getitem__``, and they need to be separately overwritten. — MisterMiyagi, Jun 03 '20 at 20:24
For your last question you need metaclasses...for this question `object` uses `type` .So you can also use `type` — vks, Jun 03 '20 at 20:25
Thanks for your comments @MisterMiyagi @vks! I have trouble wrapping my head around this though. Is what you are saying that metaclass and parent class are not the same? — mapf, Jun 03 '20 at 20:28
Yes, parent/base class and meta class are two different things. The parent/base class satisfies ``issubclass(my_class, base_class)`` – a class "is a" variant of its baseclass. The metaclass is the class' class – that is, ``isinstance(my_class, meta_class)``. A class "is an *instance of*" its metaclass. — MisterMiyagi, Jun 03 '20 at 20:33
Thank you @MisterMiyagi! I will still need some time to properly understand this, but at least this explains my error of thinking and my assuption was wrong. Still, even though these questions are not related, is there such a thing as a `UserObject` e.g.? — mapf, Jun 03 '20 at 20:42
I'm not really sure what exactly you mean by ``UserObject``, but there isn't anything that would come close. ``abc.ABC`` might be related, though it serves a different purpose. Note that you can create your own ``UserObject``, just as you could create your own ``UserList``. — MisterMiyagi, Jun 03 '20 at 20:46
@MisterMiyagi, I see, I think I get what you are saying. It would be great if you could provide an example (e.g. as an answer) though. I'll happily accept it. — mapf, Jun 03 '20 at 20:54

jsbueno · Accepted Answer · 2020-06-05T00:05:11.217

As stated in the comments - there is no such thing, and there is no need for it - all magic methods in object can be overriden and will work fine.

What happens with dictionaries, lists and other collections, as MisterMiyagi explains in the comments is that, for example, dict's get won't use the __getitem__ method, so if you are customizing the behavior of this, you also have to rewrite get. The classes that properly resolve this, allowing one to create fully working Mappings, Sequences and Sets with a minimal amount of code that is reused are the ones defined in the collections.abc module.

Now, if you want one of the magic methods to work on one object's class instead of instances of that class, you have to implement the methods in the class of the class - which is the "metaclass".

This is far different from the "superclass" - superclasses define methods and attributes the subclasses inherit, and that will be available in the subclasses. But magic methods on a class affect only the instances, not the class itself (with the exception of __init_subclass__, and, of course __new__, which can be changed to do other things than create a new instance).

Metaclasses control how classes are built (with the __new__, __init__ and __call__ methods) - and are allowed to have methods on how they behave through the magic methods - and I think there are no special cases for how the magic-methods in a metaclass work in classes, compared to what they work in the relation of a class to an ordinary instance. If you implement __add__, __getitem__, __len__ on the metaclass, all of these will work for the classes created with that metaclass.

What is possible to do, if you don't want to write your magic methods for the class in the metaclass itself, is to create a metaclass that will, on being called, automatically create another, dynamic metaclass, and copy over the dunder methods to that class. But it is tough to think of that as a "healthy" design in any serious application - having magic methods that apply to classes is already a bit over the top - although there can be cases where that is convenient.

So, for example, if you want a class to have a __geitem__ that would allow you to retrieve all instances of that class, this would work:

class InstanceRegister(type):
    def __init__(cls, name, bases, namespace, **kw):
        super().__init__(name, bases, namespace, **kw)

        cls._instances = []

        original_new = cls.__new__
        def new_wrapper(cls, *args, **kw):
            if original_new is object.__new__:
                instance = original_new(cls)
            else:
                instance = original_new(cls, *args, **kw)
            cls._instances.append(instance)
            return instance

        cls.__new__ = new_wrapper

    def __len__(cls):
        return len(cls._instances)

    def __getitem__(cls, item):
        return cls._instances[item]

And this will just work, as can be seem in this interative session:

In [26]: class A(metaclass=InstanceRegister): pass                                                                                   

In [27]: a = A()                                                                                                                     

In [28]: len(A)                                                                                                                      
Out[28]: 1

In [29]: A[0] is a                                                                                                                   
Out[29]: True

Is there an implementation of the PyObject class that allows you to properly override magic methods?

1 Answers1