Why is a type identified as not itself in mypy?

Question

MRE: https://mypy-play.net/?mypy=latest&python=3.10&gist=263dfa4914d0317291638957e51e8700&flags=strict

from typing import Callable, TypeVar, Type, Protocol

from typing_extensions import Self

_CT = TypeVar("_CT")
_T = TypeVar("_T")


class LtConverter(Protocol):
    def __new__(cls: Type[_T], x: _CT) -> _T: ...

    def __lt__(self, other: Self) -> bool: ...


def lt_through(conv: Type[LtConverter]) -> Callable[[_CT, _CT], bool]:
    def comparator(op: Callable[[LtConverter, LtConverter], bool]) -> Callable[[_CT, _CT], bool]:
        def method(self: _CT, other: _CT) -> bool:
            return op(conv(self), conv(other))
        return method
    return comparator(lambda x, y: x < y)

In the MRE, lt_through(conv) returns a function, which can be used as a method for a class, such that the "less than" operator of the class compares the class after been casted to type conv, which is any class supporting the protocol to casting and the less than operation with itself.

mypy gives the following error message:

main.py:18: error: Argument 1 to "LtConverter" has incompatible type "_CT"; expected "_CT"  [arg-type]

I am vaguely confused by this message. Why is _CT considered to be different from _CT? Why does this error occur for conv(self) but not for conv(other) (which can be seen by inspecting the char number in the verbose output)?

You probably meant to [define `__init__` instead of `__new__`](https://stackoverflow.com/questions/8106900/new-and-init-in-python)? i.e. `def __init__(self, x: _CT): ...`? Not to mention the signature for `__new__` as you have defined is incorrect. — metatoaster, May 29 '23 at 02:31
Does this answer your question? [\_\_new\_\_ and \_\_init\_\_ in Python](https://stackoverflow.com/questions/8106900/new-and-init-in-python) — Michael M., May 29 '23 at 02:33
Moreover, would have complained about `conf(other)`, just that the first one short-circuited the error somehow (if you put `c: LtConverter = conv(other)` before that, it will complain about `conv(other)` too). — metatoaster, May 29 '23 at 02:38
@Michael @metatoaster Thank you for your responses. I now understand their difference a bit better now. I initially used `__new__` because my IDE would complain about the unused parameter in the protocol. However, I am still intrigued by the behaviour in question; would you provide insight on why it doesn't work for mypy? Thanks again. — chnmasta05, May 29 '23 at 02:45
What you are asking is very much the most basic of Python classes, I encourage you to find your answers [in this thread](https://stackoverflow.com/questions/8609153/why-do-we-use-init-in-python-classes) - you are effectively initializing some `LtConverter` compatible thing by calling `conv`, and that `conv` is a `Type`, you want to pass an argument to it so that's calling `__init__`. — metatoaster, May 29 '23 at 02:49

dROOOze · Accepted Answer · 2023-05-29T04:46:31.853

Your code is almost correct - mypy's reporting just didn't do very well to allow easily fixing the remaining issues. I'll answer these two first:

I am vaguely confused by this message. Why is _CT considered to be different from _CT?

The _CT in def __new__(cls: Type[_T], x: _CT) -> _T: ... is not actually the same _CT as the one found in def method(self: _CT, other: _CT) -> bool: - you've just used the same name _CT in two different type variable contexts. Rewriting this implementation, this is what mypy actually sees you're doing:
```
_CT1 = TypeVar("_CT1")
_CT2 = TypeVar("_CT2")

...

class LtConverter(Protocol):
     def __new__(cls: Type[_T], x: _CT1) -> _T:
         ...

...

     def method(self: _CT2, other: _CT2) -> bool:
          return op(conv(self), conv(other))  # mypy: Argument 1 to "LtConverter" has incompatible type "_CT2"; expected "_CT1" [arg-type]
```

Why does this error occur for conv(self) but not for conv(other) (which can be seen by inspecting the char number in the verbose output)?

Very unfortunately, this is because mypy sometimes can't report multiple errors on one line. You'll get the second one appear if you reformat the code a bit, like the following:

      def method(self: _CT, other: _CT) -> bool:
          return op(
              conv(self),  # mypy: Argument 1 to "LtConverter" has incompatible type "_CT"; expected "_CT" [arg-type]
              conv(other),  # mypy: Argument 1 to "LtConverter" has incompatible type "_CT"; expected "_CT" [arg-type]
          )

If I understand your intention correctly, to fix this, you just have to change LtConverter.__new__ to one of the following:

def __new__(cls: Type[_T], x: typing.Any) -> _T: ...
def __new__(cls, x: typing.Any) -> Self: ...

This is based on the assumptions that

def method(self: _CT, other: _CT): means I don't care what self and other are, as long as they're up-castable to the same type, and
LtConverter is capable of being instantiated with any object to provide a valid __lt__ comparison.

In practice, your intended implementation might be a bit stricter than this, in which case you should consider putting a type variable bound on _CT, parameterising LtConverter with a _CT type, and/or replacing LtConverter.__new__::x: typing.Any with x: _CT. Here's one possible version with stricter typing:

from typing import Callable, TypeVar, Type, Protocol

from typing_extensions import Self

_CT = TypeVar("_CT")
_T = TypeVar("_T")
_ConverteeT = TypeVar("_ConverteeT", covariant=True)


class LtConverter(Protocol[_ConverteeT]):
    def __new__(cls: Type[_T], x: _ConverteeT) -> _T: ...

    def __lt__(self, other: Self) -> bool: ...


def lt_through(conv: Type[LtConverter[_CT]]) -> Callable[[_CT, _CT], bool]:
    def comparator(op: Callable[[LtConverter[_CT], LtConverter[_CT]], bool]) -> Callable[[_CT, _CT], bool]:
        def method(self: _CT, other: _CT) -> bool:
            return op(conv(self), conv(other))
        return method
    return comparator(lambda x, y: x < y)

Why is a type identified as not itself in mypy?

1 Answers1