Primitives class variables of metaclass not shared between subclasses in python, but objects are, why?

Question

consider these two pieces of code:

class M(type):
    hey = []

class S(metaclass=M):
    pass

class N(metaclass=M):
    pass

print(S.hey)  # output []
print(N.hey)  # output []
S.hey.append(6)
print(S.hey)  # output [6]
print(N.hey)  # output [6]

As you can see, changing hey list on S also modifies the hey that N has. So one can assume they are "sharing" this object whose reference is a class variable in their shared metaclass.

Now consider this version:

class M(type):
    hey = 5

class S(metaclass=M):
    pass

class N(metaclass=M):
    pass

print(S.hey)  # output 5
print(N.hey)  # output 5
S.hey = 6
print(S.hey)  # output 6
print(N.hey)  # output 5

Here we see that modifying hey for S does not affect its value when accessed through H! I have to assume that primitives behave this way.

My question is: This seems weird to me, so is there some kind of documentation on these topics that I can read to learn the "rules"? I skimmed through the official Python documentation but there is very little about the OOP side of the language. Also, why is python designed this way? What is the design deliberation behind this behavior?

EDIT: also, what do you call S and N in my code examples with respect to M? I inaccurately called them subclasses for M, but what is the correct term for them?

I was expecting consistent behavior between primitives and objects but the behavior changes

Mandatory link to [Ned Batchelder](https://nedbatchelder.com/text/names.html) — quamrana, Jul 05 '23 at 19:22
Assignment never mutates an object but creates a new reference. — mkrieger1, Jul 05 '23 at 19:22
Right. `S.hey` and `N.hey` are two separate variables, but are initially both bound to the same list object. If you modify the list object, it will be seen in both places. But if you assign a new object to `S.hey`, that doesn't affect `N.hey` -- it's still bound to that list. This is one of the KEY concepts to Python success. — Tim Roberts, Jul 05 '23 at 19:24
Unlike C#, Python does not contain primitives. Everything is an object. EVERYTHING. Strings, integers, floats, lists, dicts, sets, functions, classes -- everything is an object. Names just hold references to objects. Names never contain their value. — Tim Roberts, Jul 05 '23 at 19:29
`S.hey.append(6)` changes (mutates) whatever object `S.hey`refers to. Whatever other variable, class attribute or whatnot also references said object will be affected by that mutation. `S.hey = 6` binds/reassigns the attribute. The object previously referred to by `S.hey` (and all its references) could not care less about that. — user2390182, Jul 05 '23 at 19:36

Tim Roberts · Accepted Answer · 2023-07-05T19:32:47.990

3

You don't need classes and metaclasses to see this principle.

a = []
b = a
c = a
a.append(3)

If you now print a, b or c, you'll see [3]. That snippet contains exactly ONE list. It just has three different names. But if I do:

b = 77

That binds a new integer object to the name b. a and c are still bound to that original list, but b now has a different lifetime.

This is also the cause of a very common error:

a, b, c = [[]] * 3

It is usually a surprise to people to learn that this snippet ALSO contains exactly one list. To initialize three different lists, you need something like:

a, b, c = ([] for _ in range(3))

edited Jul 05 '23 at 19:32

answered Jul 05 '23 at 19:27

Tim Roberts

48,973
4
21
30

1

I certainly did, thank you. – Tim Roberts Jul 05 '23 at 19:32
Yes yes that's all good and fine now. follow up question though: in my code example, why doesn't Python create separate lists for each of J and N if it wants to keep them separate? – Amir Kooshky Jul 05 '23 at 19:41
Fun fact: `a, b, c = map(list, [[]] * 3)` is still shorter than the rangy generator, but it is an abomination in its own right. – user2390182 Jul 05 '23 at 19:43
@AmirKooshky How can it? `class M(type): hey = []` is a very general notation. In `[]`'s place could be any random object. How can Python know how to recreate such object (if at all possible) or make a copy of it (and if so how deep?) for every subclass? [Mutable default arguments](https://stackoverflow.com/questions/1132941/least-astonishment-and-the-mutable-default-argument) touch the same nerve. – user2390182 Jul 05 '23 at 19:45
@user2390182 You're right, there is no design behind it, thanks! – Amir Kooshky Jul 05 '23 at 19:56
*Why doesn't Python create separate lists for each of J and N if it wants to keep them separate* -- Who says we want them to be separate? You seem to be thinking this is some sort of flaw, but you should not criticize something just because you don't understand it. There are ways to make a new list if you need it, but it needs to be up to the programmer. – Tim Roberts Jul 05 '23 at 22:13

score 1 · Answer 2 · answered Jul 05 '23 at 19:54

@TimRoberts has you covered on the binding vs mutation problem. To create independent class attributes, override __new__:

class M(type):
    def __new__(cls, name, bases, dct):
        klass = super().__new__(cls, name, bases, dct)
        klass.hey = []
        return klass

Since this method is called for every new instance of the metaclass (yes, that answers your last question, classes are instances of metaclasses. That's why attributes are looked up on the metaclass if not found on the class itself, just like for any Python object).

for that effect, it is better to just write the `__init__` metaclass method than overriding `__new__`. — jsbueno, Jul 05 '23 at 20:01

score 0 · Answer 3 · answered Jul 05 '23 at 19:54

When you do S.hey.append(...) - you are modifying the existing attribute - and it is located in the class of S, that is, its metaclass.

WHen you do S.hey = 5 you are creating a new attribute in the class S, which shadows the one in the metaclass - Just like doing it in an ordinary instance would shadow the class attribute: there is nothing special regarding metaclasses to what is taking place:

class SpaceShip:
    energy = 10
    ...

class GameAction:

    def __init__(self):
        self.ship = SpaceShip()
        ...
    def took_hit(self):
        self.ship.energy -= 1
        ...

this creates a new "energy" attribute in the self.ship instance: it reads the existing value "10" which is shared across all "pristine" ship instances, subtracts one, and assigns the result back: the result is assigned in the instance, not in the class. As you can see this is a natural and useful pattern.

Back to your code, to change the shared value, you just have to explicitly assign a value to the class of S to see the result reflected in the sibling class:

S.__class__.hey = 5
N.hey
-> 5

(As for what to call the reverse relation of a "metaclass" there is no special term, but you can just say "instance class" or "class that is an instance of M". Understanding that expression however, will require that your audience is already in the context of metaclasses vs classes relationship, or confusion will ensue - the better thing is to make it very explicit: "class N created with the metaclass M". )

Primitives class variables of metaclass not shared between subclasses in python, but objects are, why?

3 Answers3