1

What's the proper way to extend a class __init__ method while keeping the type annotations intact?

Take this example class:

class Base:
    def __init__(self, *, a: str):
        pass

I would like to subclass Base and add a new parameter b to the __init__ method:

from typing import Any

class Sub(Base):
    def __init__(self, *args: Any, b: str, **kwargs: Any):
        super().__init__(*args, **kwargs)

The problem with that approach is that now Sub basically accepts anything. For example, mypy will happily accept the following:

Sub(a="", b="", invalid=1). # throws __init__() got an unexpected keyword argument 'invalid'

I also don't want to redefine a in Sub, since Base might be an external library that I don't fully control.

khelwood
  • 55,782
  • 14
  • 81
  • 108
Cesar Canassa
  • 18,659
  • 11
  • 66
  • 69
  • By copying the signature from the base class's initializer into your subclass..? – AKX Jan 10 '23 at 13:52
  • @AKX That's not ideal. I am extending an external library that I don't fully control. – Cesar Canassa Jan 10 '23 at 13:55
  • 1
    Not much you can do about that, really. You're overriding `__init__` and as far as Python and Mypy are concerned, it could be an entirely different implementation. – AKX Jan 10 '23 at 13:57
  • @CesarCanassa: The problem is that, if you don't control it, you also don't know when it might change, so any annotations will always have to chase the current implementation. On top of that, if a subclass inherits your class, and some other unrelated class (`class GrandSub(Sub, Unrelated):`), the extra arguments to `Unrelated.__init__` have to pass through `Sub.__init__` for cooperative inheritance (`Base` would also need to accept arbitrary args and pass them on to be fully compliant with that scenario), but those extra arguments have nothing to do with `Base`/`Sub`; you can't predict them. – ShadowRanger Jan 10 '23 at 13:57
  • @ShadowRanger I am not sure if I understood you correctly, but the only concern that I see is if Base suddenly changes and starts also accepting "b" which would conflict with the subclass. That's a risk I am willing to accept – Cesar Canassa Jan 10 '23 at 14:09
  • Related question: https://stackoverflow.com/q/73997582/3216427 – joanis Jan 10 '23 at 14:14
  • @joanis Suggesting dataclasses probably doesn't help if OP isn't in control of the parent class. – AKX Jan 10 '23 at 14:20
  • @CesarCanassa: If you're willing to accept that, and you're not worried about subclasses of `Sub`, then just replicate the signature in the child? The problem with properly written true cooperative multiclassing is that, by definition, you can never know all your arguments for sure. If you need to know them, and code generation facilities like `dataclasses` can't write it for you, then you can't write it "properly" no matter what, so write it improperly. – ShadowRanger Jan 10 '23 at 14:26
  • 1
    You can't do static type analysis here, because the function invoked by `super().__init__` is determined dynamically. (It's *not* necessarily `Base.__init__`, because some other class might use `Sub` in a multiple-inheritance hierarchy..) – chepner Jan 10 '23 at 14:26
  • 1
    The general advice for `__init__` in a cooperative multiple inheritance hierarchy is to not use positional arguments at all. Only use keyword arguments, and let each `__init__` "peel off" the ones it recognizes while passing the rest up the tree. – chepner Jan 10 '23 at 14:27
  • Not everything in Python can be statically typed. – chepner Jan 10 '23 at 14:27
  • @AKX that's a good point, I should have read the question and comments more carefully. dataclasses won't help at all with an external base class. I deleted that comment now. – joanis Jan 10 '23 at 15:24

2 Answers2

2

There is a solution for the question of "adding parameters to a method signature" - but it's not pretty... Using ParamSpecs and Concatenate you can essentially capture the parameters of your Base init and extend them.

Concatenate only enables adding new positional arguments though. The reasoning for that is stated in the PEP introducing the ParamSpec. In short, when adding a keyword-parameter we would run into problems if that keyword-parameter is already used by the function we're extending.

Check out this code. It's quite advanced but that way you can keep the type annotation of your Base class init without rewriting them.

from typing import Callable, Type, TypeVar, overload
from typing_extensions import ParamSpec, Concatenate


P = ParamSpec("P")

TSelf = TypeVar("TSelf")
TReturn = TypeVar("TReturn")
T0 = TypeVar("T0")
T1 = TypeVar("T1")
T2 = TypeVar("T2")

@overload
def add_args_to_signature(
   to_signature: Callable[Concatenate[TSelf, P], TReturn],
   new_arg_type: Type[T0]
) -> Callable[
   [Callable[..., TReturn]], 
   Callable[Concatenate[TSelf, T0, P], TReturn]
]:
   pass

@overload
def add_args_to_signature(
   to_signature: Callable[Concatenate[TSelf, P], TReturn],
   new_arg_type0: Type[T0],
   new_arg_type1: Type[T1],
) -> Callable[
   [Callable[..., TReturn]], 
   Callable[Concatenate[TSelf, T0, T1, P], TReturn]
]:
   pass


@overload
def add_args_to_signature(
   to_signature: Callable[Concatenate[TSelf, P], TReturn],
   new_arg_type0: Type[T0],
   new_arg_type1: Type[T1],
   new_arg_type2: Type[T2]
) -> Callable[
   [Callable[..., TReturn]], 
   Callable[Concatenate[TSelf, T0, T1, P], TReturn]
]:
   pass

# repeat if you want to enable adding more parameters...

def add_args_to_signature(
   *_, **__
):
   return lambda f: f

class Base:
   def __init__(self, some_arg: float, *, some_kwarg: int):
      pass


class Sub(Base):

   # Note: you'll lose the name of your new args in your code editor. 
   @add_args_to_signature(Base.__init__, str)
   def __init__(self, you_can_only_add_positional_args: str, /, *args, **kwargs):
      super().__init__(*args, **kwargs)
   

Sub("hello", 3.5, some_kwarg=5)

VS-Code gives the following type hints for Sub: Sub(str, some_arg: float, *, some_kwarg: int)

I don't know though if mypy works with ParamSpec and Concatenate...

Due to a bug in VS-Code the position of the parameters aren't correctly matched though (the parameter being set is off by one).

Note that this is quite an advanced use of the typing module. You can ping me in the comments if you need additional explanations.

Robin Gugel
  • 853
  • 3
  • 8
  • *...quite an advanced abuse* no, it's a perfectly valid use case without tricks or `mypy`-specific treatment, so this is intended usage. – STerliakov Jan 10 '23 at 16:42
  • @SUTerliakov Yeah I guess you're right... I updated _advanced abuse_ to _advanced use_ – Robin Gugel Jan 10 '23 at 16:49
-1

The real answer depends on your use case. You can (for instance) implement the __new__() special method on your class to create an object of a given type dependent on the parameter provided to __init__().

In most cases this is rather too complicated and Python provides a generic means for this that covers a lot of use cases. Take a look at Python DataClasses

The other way might be to review the more general response to this question that I wrote some time back. "SubClassing Int in Python", specifically the example of a Modified Type class.

Jay M
  • 3,736
  • 1
  • 24
  • 33
  • Also, Python does support Generic classes but I've not used them. https://docs.python.org/3/library/typing.html see typing.Generic – Jay M Jan 10 '23 at 15:38
  • -1, because this doesn't answer the question (and I can't even imagine a question such that this answer fits it). Though the linked answer is beautiful. The question was specifically about type hinting of overridden methods in subclass (not sure why esp. `__init__` was chosen), and none of our links touches that topic. – STerliakov Jan 10 '23 at 16:41