Generic NamedTuple in Python 3.6

Question

I'm trying to create a generic version of a NamedTuple, as follows:

T1 = TypeVar("T1")
T2 = TypeVar("T2")

class Group(NamedTuple, Generic[T1, T2]):
    key: T1
    group: List[T2]

g = Group(1, [""])  # expecting type to be Group[int, str]

However, I get the following error:

TypeError: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases

I'm not sure how else to achieve what I'm trying to do here, or if this might be a bug in the typing mechanism on some level.

You may further elaborate, in which respect you intend to generalize NamedTuple, which is pretty general in my opinion. From your code snippet I can't recognize... — guidot, May 25 '18 at 14:12
The special `NamedTuple` doesn't support *any other base classes*. Full stop. The `NamedTuple` base class delegates all use to a metaclass that will run `collections.namedtuple()` with a small amount of customisation, which means that the resulting class *only ever inherits from `tuple`*. This is a wider problem with `NamedTuple`, not limited to `Generic`. — Martijn Pieters, Sep 28 '19 at 15:12

FHTMitchell · Accepted Answer · 2021-04-23T16:09:18.840

18

So this is a metaclass conflict since in python 3.6 the typing NamedTuple and Generic use different metaclasses (typing.NamedTupleMeta and typing.GenericMeta), which python can't handle. I'm afraid there is no solution to this, other than to subclass from tuple and manually initialise the values:

T1 = TypeVar("T1")
T2 = TypeVar("T2")

class Group(tuple, Generic[T1, T2]):

    key: T1
    group: List[T2]

    def __new__(cls, key: T1, group: List[T2]):
        self = tuple.__new__(cls, (key, group))
        self.key = key
        self.group = group
        return self            

    def __repr__(self) -> str:
        return f'Group(key={self.key}, group={self.group})'

Group(1, [""])  # --> Group(key=1, group=[""])

Due to PEPs 560 and 563 this is fixed in python 3.7:

Python 3.7.0b2 (v3.7.0b2:b0ef5c979b, Feb 28 2018, 02:24:20) [MSC v.1912 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from __future__ import annotations
>>> from typing import *
>>> T1 = TypeVar("T1")
>>> T2 = TypeVar("T2")
>>> class Group(NamedTuple, Generic[T1, T2]):
...     key: T1
...     group: List[T2]
...
>>> g: Group[int, str] = Group(1, [""])
>>> g
Group(key=1, group=[''])

Of course in python 3.7 you can just use a dataclass which are less lightweight (and mutable) but serve similar purposes.

from dataclasses import dataclass, astuple
from typing import Generic, TypeVar, List

T1 = TypeVar('T1')
T2 = TypeVar('T2')

@dataclass
class Group(Generic[T1, T2]):

     # this stores the data like a tuple, but isn't required
     __slots__ = ("key", "group")

     key: T1
     group: List[T2]

     # if you want to be able to unpack like a tuple...
     def __iter__(self):
          yield from astuple(self)


g: Group[int, str] = Group(1, ['hello', 'world'])
k, v = g
print(g)

How well type checkers handle my solution / yours in python 3.7 though I haven't checked. I suspect it may not be seamless.

Edit

I found another solution -- make a new metaclass

import typing
from typing import *

class NamedTupleGenericMeta(typing.NamedTupleMeta, typing.GenericMeta):
    pass


class Group(NamedTuple, Generic[T1,T2], metaclass=NamedTupleGenericMeta):

    key: T1
    group: List[T2]


Group(1, ['']) # --> Group(key=1, group=[''])

edited Apr 23 '21 at 16:09

answered May 25 '18 at 14:14

FHTMitchell

11,793
2
35
47

3

The second solution looks nice, but unfortunately it doesn't fully work for type hinting:`g: Group[int, str] = Group(1, [""]) TypeError: 'type' object is not subscriptable` – pdowling May 25 '18 at 15:40
Ah ok. What if you swap the metaclasses bases around? How did the first one go? – FHTMitchell May 25 '18 at 15:42
2

You still can't *use* the `Generic[...]` hint in `NamedTuple` subclasses, because due to the very same metaclass conflict the newly generated named tuple class won't have the `__class_getitem__` hook needed to make concrete hints like `Group[str, int]` work. That's because the `NamedTuple` metaclass returns a new class object with only `tuple` as a base class, not `Generic`, and the requisite `__parameters__` attribute on the class that records the available typevars is completely gone. – Martijn Pieters Sep 28 '19 at 14:57
@MartijnPieters fair point, but fixed with `from __future__ import annotations` – FHTMitchell Feb 04 '20 at 13:35
1

Please make that more explicit in your answer, because you need that compiler switch in *all* modules that use that subclass. – Martijn Pieters Feb 04 '20 at 14:16
also mypy complains "Generic tuple types not supported" – Anentropic Sep 17 '20 at 14:05
1

and with the dataclass + `__iter__` method workaround, when you do a destructuring assignment the resulting vars are `Any` typed due to limitation of `astuple` (this issue https://github.com/python/mypy/issues/5152) But I guess this is the best we can do for the foreseeable future... the relevant issue for supporting generic namedtuples in mypy has been open and unfixed for 5 years now: https://github.com/python/mypy/issues/685 – Anentropic Sep 17 '20 at 18:19
downvoting because the first option doesn't retain the immutability of `NamedTuple` and the third option doesn't retain its type safety when unpacking – joel Oct 22 '20 at 16:25
19

In python 3.9, OP's original code now breaks on declaration of the class, even if the class goes unused, with the error `TypeError: Multiple inheritance with NamedTuple is not supported` – cowlinator Apr 22 '21 at 01:54
2

Also, on Python 3.7+, `typing.GenericMeta` doesn't seem to exist. https://github.com/mkdocstrings/mkdocstrings/issues/2 . – cowlinator Apr 22 '21 at 02:15
yup, seems they broke it. My suggestion is to use a `dataclass(frozen=True)` with `__slots__` if you need them. Still, so annoying! – FHTMitchell Apr 23 '21 at 16:06

Generic NamedTuple in Python 3.6

1 Answers1

Edit

Linked

Related