64

I have some classes looking like this:

class Base:
  subs = [Sub3,Sub1]
  # Note that this is NOT a list of all subclasses!
  # Order is also important

class Sub1(Base): pass
class Sub2(Base): pass
class Sub3(Base): pass
...

Now, this fails because Sub1 and Sub3 are not defined when Base.subs is. But obviously I can't put the subclasses before Base either. Is there a way to forward-declare classes in Python? I want to work with isinstance so the types in subs actually have to be the same as the later declared subclasses, it's not enough that they have the same name and other properties.

One workaround is to do: Base.subs = [Sub3,Sub1] after the subclasses have been defined, but I don't like having to split my class in that way.

Edit: Added information about order

martineau
  • 119,623
  • 25
  • 170
  • 301
pafcu
  • 7,808
  • 12
  • 42
  • 55
  • 7
    This is a bad design. You're conflating a factory (which is aware of subclasses) with the superclass (which does not need to be aware of subclasses). Why do this? Why not simply separate things and make things simpler? – S.Lott Nov 12 '10 at 11:03
  • 1
    @S.Lott: What if it does not actually _need_ to be aware of it's subclasses per se. It just needs to have a bunch of classes in a list, some of which might be its subclasses. – pafcu Nov 12 '10 at 11:44
  • 2
    @pafcu: "some of which might be its subclasses". This is still a bad idea -- a superclass should never know about it's subclasses. Doing so violates a principle of OO design. You can no longer simply create new subclasses without also changing the superclass. You have to separate the "list of classes" from the superclass. The only reason for having a "list of classes" is to create a factory. A factory is a good thing; it's not a feature of a superclass, however. – S.Lott Nov 12 '10 at 13:13
  • What's the purpose of the `subs` list, or more specifically, what determines whether and in what order the subclasses get put in it? @S.Lott's critique *may* be valid depending on exactly what you're trying to achieve, but more information is required to actually make such a judgment. Depending on what that is, it might be possible to make the base class and subclasses cooperate without having to modify the base class every time you add a subclass -- a weakness of the approach you're now using. – martineau Nov 12 '10 at 17:50
  • 53
    I appreciate the help, and I don't want to seem ungrateful, but I can not help but wonder why people on SO seem to spend so much time thinking about motives and reasons behind questions instead of just answering the questions. It's a fairly straightforward question after all. – pafcu Nov 12 '10 at 18:01
  • 10
    @pafcu: If you start down the wrong road and write code with a poor design, we can't really help you except to say, stop doing that, and start down a road that doesn't have the obvious problems. It's not a "straightforward question". If it was, you would have found the answer on your own. The question reflects a common design mistake. Fix the mistake and you no longer need to ask this question. – S.Lott Nov 12 '10 at 21:17
  • 2
    @pafcu: The reason is usually so they can provide you with better answer -- and perhaps even tell or show you how to avoid the problem altogether. Some are better at communicating this than others, however. A good analogy might be the difference between just giving someone sick some medicine verses showing them how to avoid getting ill in the first place. – martineau Nov 13 '10 at 01:22
  • 14
    @martineau: This is a developer community, right? It's like a doctor asking another doctor for medicine, usually he is not second guessed unless he is _obviously_ wrong. I have a reason to ask this specific question. The reason might not be clear due to the limited information provided by me, but it still exists. There is no reason to believe there is _no_ case ever when my question would be valid. Anyway, this is getting offtopic so enough about this. – pafcu Nov 13 '10 at 09:04
  • @pafcu: It's reasonable, when someone asks how to do something unusual here for others to wonder why or question the design. Personally I try to avoid judgments until I have enough information to make them. In this case I happen to know of a least one perfectly valid and powerful design where the base class needs to all the subclasses so it can implement a class factory itself, although I don't think that's the only possible use case. However, from what little you've told us, that doesn't sound like what you're doing and I like to know more for my own enlightenment, not so I can put you down. – martineau Nov 13 '10 at 22:50
  • 20
    @martineau: Yes, I understand people are curious (I am often too, and it's a great way to learn), but SO is a questions/answers site, and (IMHO) not meant to to discuss design choices in-depth (unless the question is specifically about this). I apologize for sounding aggressive, but it just seems that when I ask a question here I end up spending more time explaining my design choices than I would have used to solve the problem on my own. – pafcu Nov 14 '10 at 22:41
  • 4
    Have to agree. "Why are you doing this?" is not the least helpful. For example, I came to this page because I wanted to understand forward referencing in Python, not to read a discussion of code design. We should answer the question and then question the design if we cannot contain ourselves. – Ray Salemi Nov 17 '20 at 13:35
  • The problem is that `subs = [Sub3,Sub1]` is *code that runs while creating* `Base`, so even if Python had a way to "know" that `Sub3` and `Sub1` are the names of classes at this point, it has to actually *have* those classes to put into the list. – Karl Knechtel Sep 02 '22 at 06:19

11 Answers11

26

Here's essentially a hybrid version of @Ignacio Vazquez-Abrams' and @aaronasterling's answers which preserves the order of the subclasses in the list. Initially the desired subclass names (i.e. strings) are manually placed in the subs list in the desired order, then as each subclass is defined, a class decorator causes the corresponding string to be replaced with the actual subclass:

class Base(object):  # New-style class (i.e. explicitly derived from object).

    @classmethod
    def register_subclass(cls, subclass):
        """ Class decorator for registering subclasses. """

        # Replace any occurrences of the class name in the class' subs list.
        # with the class itself.
        # Assumes the classes in the list are all subclasses of this one.
        # Works because class decorators are called *after* the decorated class'
        # initial creation.
        while subclass.__name__ in cls.subs:
            cls.subs[cls.subs.index(subclass.__name__)] = subclass

        return cls  # Return modified class.

    subs = ['Sub3', 'Sub1']  # Ordered list of subclass names.


@Base.register_subclass
class Sub1(Base): pass

@Base.register_subclass
class Sub2(Base): pass

@Base.register_subclass
class Sub3(Base): pass

print('Base.subs: {}'.format(Base.subs))
# Base.subs: [<class '__main__.Sub3'>, <class '__main__.Sub1'>]

Update: Metaclasses

Exactly the same thing can also be done using a metaclass—which has the advantage that it eliminates the need to explicitly decorate each subclass as shown in my original answer above (which you accepted), however it makes it all happen automagically.

Note that even though the metaclass' __init__() is called for the creation of every subclass, it only updates the subs list if the subclass' name appears in it—so the initial Base class' definition of the contents of the subs list still controls what gets replaced in it (maintaining its order).

class BaseMeta(type):

    def __init__(cls, name, bases, classdict):
        if classdict.get('__metaclass__') is not BaseMeta:  # Metaclass instance?
            # Replace any occurrences of a subclass' name in the class being
            # created the class' sub list with the subclass itself.
            # Names of classes which aren't direct subclasses will be ignored.
            while name in cls.subs:
                cls.subs[cls.subs.index(name)] = cls

        # Chain to __init__() of the class instance being created after changes.
        # Note class instance being defined must be new-style class.
        super(BaseMeta, cls).__init__(name, bases, classdict)


# Python 2 metaclass syntax.
class Base(object):  # New-style class (derived from built-in object class).
    __metaclass__ = BaseMeta
    subs = ['Sub3', 'Sub1']  # Ordered list of subclass names.

# Python 3 metaclass syntax.
#class Base(metaclass=BaseMeta):
#    subs = ['Sub3', 'Sub1']  # Ordered list of subclass names.


# Note: No need to manually register the (direct) subclasses.
class Sub1(Base): pass
class Sub2(Base): pass
class Sub3(Base): pass

print('Base.subs: {}'.format(Base.subs))

Output:

Base.subs: [<class '__main__.Sub3'>, <class '__main__.Sub1'>]

It important to note that there's at least one subtle difference between these two answers—namely that the first will work with any class name that is registered via @Base.register_subclass(), whether or not its actually a subclass of Base (although that might be possible to change/fix.)

I'm pointing this out for a couple of reasons: First because in your comments you said that subs was a "bunch of classes in a list, some of which might be its subclasses", and more importantly, because that's not the case with the code in my update, which only works for Base subclasses since they effectively get "registered" automatically via the metaclass—but will leave anything else in the list alone. This could be considered a bug or a feature. ;¬)

Update: Python 3.6+

In Python 3.6 a new object method was added named __init_subclass__() which provides an even simpler way implement things that also doesn't require decorating all the subclasses or defining a metaclass:

#!/usr/bin/env python3.6

class Base:
    subs = ['Sub3', 'Sub1']  # Ordered list of subclass names.

    def __init_subclass__(cls, **kwargs):
        super().__init_subclass__(**kwargs)
        while cls.__name__ in cls.subs:
            cls.subs[cls.subs.index(cls.__name__)] = cls


# Note: No need to manually register the subclasses.
class Sub1(Base): pass
class Sub2(Base): pass
class Sub3(Base): pass

print('Base.subs: {}'.format(Base.subs))
martineau
  • 119,623
  • 25
  • 170
  • 301
  • great answer. I wander if it works when "subclasses" are not, i.e. not derived from the base class. It seems a quite useful use case where there is a list of independent classes and an aggregator/manager class – davka Apr 04 '18 at 07:52
  • 1
    @davka: Yes, it could be used for that purpose—at least the code in the first snippet could because it doesn't care if the class being registered via `Base.register_subclass()` decorator is actually a subclass of the `Base` class or not. However, in the case of the code in update, only actual subclass names in the base class `subs` list will get replaced with the corresponding subclass (assuming one with a given name gets defined) since there's no way provided by it to force that to happen (such as via a decorator call). – martineau Apr 04 '18 at 14:31
14

Write a decorator that adds it to the registry in Base.

class Base(object):
  subs = []

  @classmethod
  def addsub(cls, scls):
    cls.subs.append(scls)

 ...

@Base.addsub
class Sub1(Base):
  pass

class Sub2(Base):
  pass

@Base.addsub
class Sub3(Base):
  pass
Ignacio Vazquez-Abrams
  • 776,304
  • 153
  • 1,341
  • 1,358
  • Sorry, forgot to specify that order is important (it's a list of "preferred" types). Having this depend on the order of the code is pretty dangerous. I guess I could add a parameter to addsub for that though. This still has the problem though that the contents of Base.subs is defined all over the place. – pafcu Nov 12 '10 at 07:39
4

Edit: Because of the added requirement of order I completely reworked my answer. I also make use of a class decorator, which was used here first by @Ignacio Vazquez-Abrams.

Edit2: code now tested and some stupidity slightly corrected


class Base(object):
    subs = []

    @classmethod
    def addsub(cls, before=None): 
        def inner(subclass):
            if before and before in cls.subs:
                cls.subs.insert(cls.subs.index(before), subclass)
            else:
                cls.subs.append(subclass)
            return subclass
        return inner

@Base.addsub()
class Sub1(Base):
    pass

class Sub2(Base):
    pass

@Base.addsub(before=Sub1)
class Sub3(Base):
    pass

knitti
  • 6,817
  • 31
  • 42
  • In that case it's hard to determine the order of the list Base.subs. It would depend on the order in which I write the code which is pretty error-prone. Also, Base.subs is a property of the Base class so I would prefer to define it there instead of spreading it out over several classes. – pafcu Nov 12 '10 at 07:36
  • Yes, and the code breaks in interesting ways if someone does not realize that class order matters (it usually doesn't). It's easy to accidentally change order when e.g. refactoring or cleaning up the code. – pafcu Nov 12 '10 at 07:46
  • I haven't seen a problem where it **should** matter. Perhaps you should think about a) being way more explicit in generating and maintaining order or b) not depending on order at all. Why should it matter? – knitti Nov 12 '10 at 07:59
  • Also, `Sub.__new__` affects, instance creation, not class creation which is what's needed here. – aaronasterling Nov 12 '10 at 08:06
  • I like this answer because the info about subclasses is not duplicated, and the base class does not know about its subclasses, which is good. The decision of whether or not and where to add a new subclass to the priority list is up to its implementer, and is in one place. A con is that the order imposed by "before" is partial, so the result depends on the order in which the classes are written (or processed) – davka Apr 04 '18 at 07:41
3

There is no way to directly declare forward-references in Python, but there are several workarounds, a few of which are reasonable:

1) Add the subclasses manually after they are defined.

    - Pros: easy to do; Base.subs is updated in one place
    - Cons: easy to forget (plus you don't want to do it this way)

Example:

class Base(object):
    pass

class Sub1(Base):
    pass

class Sub2(Base):
    pass

class Sub3(Base):
    pass

Base.subs = [sub3, sub1]

2) Create Base.subs with str values, and use a class decorator to substitute the actual subclasses (this can be a class method on Base or a function -- I'm showing the function version, although I would probably use the method version).

- Pros: easy to do
- Cons: somewhat easy to forget; 

Example:

def register_with_Base(cls):
    name = cls.__name__
    index = Base.subs.index(name)
    Base.subs[index] = cls
    return cls

class Base(object):
    subs = ['Sub3', 'Sub1']

@register_with_Base
class Sub1(Base):
    pass

class Sub2(Base):
    pass

@register_with_Base
class Sub3(Base):
    pass

3) Create Base.subs with str values, and have the method that uses Base.subs do the substitution.

- Pros: no extra work in decorators, no forgetting to update `subs` later
- Cons: small amount of extra work when accessing `subs`

Example:

class Base(object):
    subs = ['Sub3', 'Sub1']
    def select_sub(self, criteria):
        for sub in self.subs:
            sub = globals()[sub]
            if #sub matches criteria#:
                break
        else:
            # use a default, raise an exception, whatever
        # use sub, which is the class defined below

class Sub1(Base):
    pass

class Sub2(Base):
    pass

class Sub3(Base):
    pass

I would use option 3 myself, as it keeps the functionality and the data all in one place. The only thing you have to do is keep subs up to date (and write the appropriate subclasses, of course).

Ethan Furman
  • 63,992
  • 20
  • 159
  • 237
2

I'm pretty sure this should work for you. Just assign the depended class attribute afterwards. This is also a lot less complicated.

class Base:pass

class Sub1(Base): pass
class Sub2(Base): pass
class Sub3(Base): pass

Base.subs = [Sub3,Sub1]
print(Sub1.subs)
#[<class __main__.Sub3 at 0x0282B2D0>, <class __main__.Sub1 at 0x01C79810>]
Garrett Hyde
  • 5,409
  • 8
  • 49
  • 55
2

Not a direct answer to your question about forward declarations, but a direct solution for your problem of knowing all sub-classes in the base class itself: just use the (probably not very well known) __subclasses__ function, which should be defined on all classes in Python3. It directly gives you exactly what you want:

In [1]: class Base: pass

In [2]: class Sub1(Base): pass

In [3]: class Sub2(Base): pass

In [4]: class Sub3(Base): pass
    
In [5]: Base.__subclasses__()
Out[5]: [__main__.Sub1, __main__.Sub2, __main__.Sub3]

The documentation doesn't specify if the order in which the sub-classes are defined is maintained, but it does appear to do so.

See also this answer to a question that asks about your problem.

Bas Swinckels
  • 18,095
  • 3
  • 45
  • 62
1

I would just define the subclasses as strings and have the inevitable decorator replace the strings with the classes that they name. I would also define the decorator on a metaclass because I think that that's more in line with the objective: we're modifying class behavior and just like you modify object behavior by modifying its class, you modify class behavior by modifying its metaclass.

class BaseMeta(type):

    def register(cls, subcls):
        try:
            i = cls.subs.index(subcls.__name__)
        except ValueError:
            pass
        else:
            cls.subs[i] = subcls
        finally:
            return cls


class Base(object):
    __metaclass__ = BaseMeta
    subs = ['Sub3', 'Sub1']

@Base.register
class Sub1(Base): pass

@Base.register
class Sub2(Base): pass

@Base.register
class Sub3(Base): pass

print Base.subs

This outputs:

[<class '__main__.Sub3'>, <class '__main__.Sub1'>]
aaronasterling
  • 68,820
  • 20
  • 127
  • 125
  • This seems like a really interesting approach because I can specify the elements of base.subs all in one place. Makes it much easier to change if needed. – pafcu Nov 12 '10 at 09:13
  • @martineau, The point is that we are changing the class's behavior. Doing this by changing the class itself is essentially monkey patching an object. The reason to change a class is to change the behavior of its instances - not to change its own behavior. You do that by changing the class of which _it_ is an instance, namely its metaclass. We're not programming in C++ anymore, classes are objects too :) Also, I deleted my snarky comment on your answer. I haven't actually been seeing comments directed to me for some reason. – aaronasterling Nov 13 '10 at 08:39
  • @aaronasterling: Thanks for the explanation. While I understand and would normally completely agree with your rationale, I don't think it really applies here because sticking the subclass decorator into the metaclass won't have any effect unless it's explicitly applied to a subclass using the `@` syntax. In other word, in this case, putting it in a metaclass just seems like a needless and potential confusing complication -- and why I consciously chose to just make it a classmethod of the Base class in my own answer. – martineau Nov 15 '10 at 16:03
  • @martineau, well I guess I've never been one for writing code that lazy people can understand - If they can great, if not oh well. So aside from the issue of confusion, my point stands. class methods are just monkey patching with pretty syntax. changing the metaclass is the right way. – aaronasterling Nov 15 '10 at 19:31
  • @aaronasterling: Well I certainly don't think I'm lazy about such matters, so while I could easily understand *how* your code worked, I wasn't sure *why* you did it that way (so I asked). From what you write it sounds like you think **A.** all classmethods are monkey patches (and therefore generally a poor way to do things) and that **B.** all class decorators are essentially just classmethods and should be put in a metaclass. Correct? I also can't help but wonder how the adage to always do "the simplest thing that could possibly work" fits into all this... – martineau Nov 16 '10 at 01:43
  • @mertineau, I wasn't calling you lazy, just saying that how isn't that hard to understand. __A__ I am currently of the opinion that class methods are monkey patches and belong as regular methods on the metaclass. __B__ decorators that happen to be methods belong on the metaclass __iff__ they affect the behavior of the class and not just the instance on which they are called. As far as the adage goes, I __A__ don't think it's a great one and __B__ think that this is simpler because it reduces things back to classes, instances and methods instead of classes, instances, class methods ... – aaronasterling Nov 16 '10 at 04:32
  • ... and regular methods. You just have to remember that classes are instances of their metaclass (which is just a class/object). – aaronasterling Nov 16 '10 at 04:34
  • Using a metaclass for such a trivial problem is like using an ORM to track a handful of simple dictionaries. – Ethan Furman Jan 19 '13 at 00:35
1

There is a workaround: put referenced dummy Sub1, Sub3 classes at the top, this works as "forward declaration". They will be replaced by the actual implementations with the same names when executed.

forward-declaration.py:

class Sub1(): 
    print("Sub1 dummy class called")
    pass
class Sub3():
    print("Sub3 dummy class called")
    pass

class Base:
  subs = [Sub3, Sub1]
  print("Base class called")

class Sub1(Base): 
    print("Sub1 class called")
    def __init__(self):
        print("Sub1:__init__ called")
    pass
class Sub2(Base):
    def __init__(self):
        print("Sub2:__init__ called")
    pass
class Sub3(Base):
    def __init__(self):
        print("Sub3:__init__ called")
    pass

sub_1 = Sub1()
sub_2 = Sub2()
sub_3 = Sub3()

print(Base.subs)

python forward-declaration.py

Sub1 dummy class called
Sub3 dummy class called
Base class called
Sub1 class called
Sub1:__init__ called
Sub2:__init__ called
Sub3:__init__ called
[<class '__main__.Sub3'>, <class '__main__.Sub1'>]

Note: The above method fails on mypy or pylint static check but works properly

Jonathan L
  • 9,552
  • 4
  • 49
  • 38
0
class Foo:
  pass

class Bar:
  pass

Foo.m = Bar()
Bar.m = Foo()
ceth
  • 44,198
  • 62
  • 180
  • 289
0

With lots classes I need some way to prototype the name of class before it is defined. Perhaps older versions of Python did not have an option, but found this...

import typing 

B = typing.NewType("B", None)

class A:
  def DoSomething(self, b:B):
    print(f"b={b}")

class B:
  def __str__(self):
      return f"this is B"

a = A()
b = B()
a.DoSomething(b)
Gabe Halsmer
  • 808
  • 1
  • 9
  • 18
-1

If local inner classes inside an outer class can fullfill the given demands, a simple solution can help (so the inner classes are declared "in forward"; at least before they are instantiated inside the outer class):

class Base:
    class Sub1:
        # ...
    class Sub2:
        # ...
    class Sub3:
        # ...

    __Object1 = Sub1()
    __Object2 = Sub2()
    __Object3 = Sub3()
    # ...