typing.NamedTuple and mutable default arguments

Question

Given I want to properly using type annotations for named tuples from the typing module:

from typing import NamedTuple, List

class Foo(NamedTuple):
    my_list: List[int] = []

foo1 = Foo()
foo1.my_list.append(42)

foo2 = Foo()
print(foo2.my_list)  # prints [42]

What is the best or cleanest ways to avoid the mutable default value misery in Python? I have a few ideas, but nothing really seems to be good

1. Using None as default

   class Foo(NamedTuple):
       my_list: Optional[List[int]] = None
   
   foo1 = Foo()
   if foo1.my_list is None
     foo1 = foo1._replace(my_list=[])  # super ugly
   foo1.my_list.append(42)
   

2. Overwriting __new__ or __init__ won't work:

   AttributeError: Cannot overwrite NamedTuple attribute __init__
   AttributeError: Cannot overwrite NamedTuple attribute __new__
   

3. Special @classmethod

   class Foo(NamedTuple):
       my_list: List[int] = []
   
       @classmethod
       def use_me_instead(cls, my_list=None):
          if not my_list:
              my_list = []
          return cls(my_list)
   
   foo1 = Foo.use_me_instead()
   foo1.my_list.append(42)  # works!
   

4. Maybe using frozenset and avoid mutable attributes altogether? But that won't work with Dicts as there are no frozendicts.

Does anyone have a good answer?

Answer 1

Use a dataclass instead of a named tuple. A dataclass allows a field to specify a default factory rather than a single default value.

from dataclasses import dataclass, field


@dataclass(frozen=True)
class Foo:
    my_list: List[int] = field(default_factory=list)

Answer 2

EDIT: Updated to use the approach of Alex, because this works much better than my previous idea.

Here is a Alex's Foo class put into a decorator:

from typing import NamedTuple, List, Callable, TypeVar, cast, Type
T = TypeVar('T')

def default_factory(**factory_kw: Callable) -> Callable[[Type[T]], Type[T]]:
    def wrapper(wcls:  Type[T]) -> Type[T]:
        def du_new(cls: Type[T], **kwargs) -> T:
            for key, factory in factory_kw.items():
                if key not in kwargs:
                    kwargs[key] = factory()
            return super(cls, cls).__new__(cls, **kwargs)  # type: ignore[misc]
        return type(f'{wcls.__name__}_', (wcls, ), {'__new__': du_new})
    return wrapper

@default_factory(my_list=list)
class Foo(NamedTuple):
    my_list: List[int] = []  # you still need to define the default argument

foo1 = Foo()
foo1.my_list.append(42)

foo2 = Foo()
print(foo2.my_list)  # prints []
#reveal_type(foo2) # prints Tuple[builtins.list[builtins.int], fallback=foo.Foo]

Answer 3

EDIT:

Blending my approach with Sebastian Wagner's idea of using a decorator, we can achieve something like this:

from typing import NamedTuple, List, Callable, TypeVar, Type, Any, cast
from functools import wraps

T = TypeVar('T')

def default_factory(**factory_kw: Callable[[], Any]) -> Callable[[Type[T]], Type[T]]:
    def wrapper(wcls: Type[T], /) -> Type[T]:
        @wraps(wcls.__new__)
        def __new__(cls: Type[T], *args: Any, **kwargs: Any) -> T:
            for key, factory in factory_kw.items():
                kwargs.setdefault(key, factory())
            new = super(cls, cls).__new__(cls, *args, **kwargs) # type: ignore[misc]
            # This call to cast() is necessary if you run MyPy with the --strict argument
            return cast(T, new)
        cls_name = wcls.__name__
        wcls.__name__ = wcls.__qualname__ = f'_{cls_name}'
        return type(cls_name, (wcls, ), {'__new__': __new__, '__slots__': ()})
    return wrapper

@default_factory(my_list=list)
class Foo(NamedTuple):
    # You do not *need* to have the default value in the class body,
    # but it makes MyPy a lot happier
    my_list: List[int] = [] 
    
foo1 = Foo()
foo1.my_list.append(42)

foo2 = Foo()
print(f'foo1 list: {foo1.my_list}')     # prints [42]
print(f'foo2 list: {foo2.my_list}')     # prints []
print(Foo)                              # prints <class '__main__.Foo'>
print(Foo.__mro__)                      # prints (<class '__main__.Foo'>, <class '__main__._Foo'>, <class 'tuple'>, <class 'object'>)
from inspect import signature
print(signature(Foo.__new__))           # prints (_cls, my_list: List[int] = [])

Run it through MyPy, and MyPy informs us that the revealed type of foo1 and foo2 is still "Tuple[builtins.list[builtins.int], fallback=__main__.Foo]"

Original answer below.

---

How about this? (Inspired by this answer here):

from typing import NamedTuple, List, Optional, TypeVar, Type

class _Foo(NamedTuple):
    my_list: List[int]


T = TypeVar('T', bound="Foo")


class Foo(_Foo):
    "A namedtuple defined as `_Foo(mylist)`, with a default value of `[]`"
    __slots__ = ()

    def __new__(cls: Type[T], mylist: Optional[List[int]] = None) -> T:
        mylist = [] if mylist is None else mylist
        return super().__new__(cls, mylist)  # type: ignore


f, g = Foo(), Foo()
print(isinstance(f, Foo))  # prints "True"
print(isinstance(f, _Foo))  # prints "True"
print(f.mylist is g.mylist)  # prints "False"

Run it through MyPy and the revealed type of f and g will be: "Tuple[builtins.list[builtins.int], fallback=__main__.Foo]".

I'm not sure why I had to add the # type: ignore to get MyPy to stop complaining — if anybody can enlighten me on that, I'd be interested. Seems to work fine at runtime.