Type hints when representing dict keys by attribute notation in python 3.6

Question

Given the example of the first answer in Accessing dict keys like an attribute?:

class AttrDict(dict):
    def __init__(self, *args, **kwargs):
        super(AttrDict, self).__init__(*args, **kwargs)
        self.__dict__ = self

and a function that returns:

def dict_to_attrdict(somedict):
    return AttrDict(**somedict)

assigned as:

data = dict_to_attrdict(mydict)

What is the correct way to add type hints for the class and function that will pass mypy checking given the following constraints:

the dict keys will always be a str
the dict values will have to be dynamic and represented by Any as they vary that I don't wish to type each individually i.e. some str, List[dict[str, List]], Dict[str, str], Dict[str, List]

score 2 · Accepted Answer · answered Jul 05 '18 at 17:32

You can make the class and function definitions themselves typecheck by doing this:

from typing import Dict, Any

class AttrDict(Dict[str, Any]):
    def __init__(self, *args: Any, **kwargs: Any) -> None:
        super(AttrDict, self).__init__(*args, **kwargs)
        self.__dict__ = self

def dict_to_attrdict(some: Dict[str, Any]) -> AttrDict:
    return AttrDict(**some)

Inheriting from Dict[X, Y] is no different from inheriting from just dict at runtime, but it gives mypy the extra metadata it needs.

However, you will not actually be able to use an instance of AttrDict in a typesafe way: mypy will always flag things like my_attrdict.foo as being an error.

This is because it's impossible to determine statically what fields will be present within AttrDict in all cases -- mypy has no idea what exactly lives inside AttrDict. And since mypy can't tell whether or not doing things like my_attrdict.foo is actually safe, it leans towards the conservative side and just decides to consider that unsafe.

You have two different options for working around this. First, if you genuinely want to keep AttrDict as dynamic as possible, you can tell mypy to just assume that the type is any arbitrary dynamic type, like so:

from typing import Dict, Any, TYPE_CHECKING

if TYPE_CHECKING:
    AttrDict = Any
else:
    class AttrDict(dict):
        def __init__(self, *args, **kwargs) -> None:
            super(AttrDict, self).__init__(*args, **kwargs)
            self.__dict__ = self

def dict_to_attrdict(some: Dict[str, Any]) -> AttrDict:
    return AttrDict(**some)

TYPE_CHECKING is a value that is always False at runtime, but is treated as being always True by mypy. The net effect is that mypy will only consider the 'if' branch of that if/else and ignore whatever's in the 'else' branch: we've now taught mypy that AttrDict is a type alias for Any: is exactly equivalent to Any. However, at runtime, we always fall into the 'else' branch and define the class like before.

The main downside of this approach is that we've really gained no value from using static typing. We're able to add a little bit of safety to dict_to_attrdict since we can now enforce that the keys must be strings, but that's it.

The second option is to lean into the strengths of mypy and rewrite your code to actually use classes. So, we'd get rid of AttrDict and actually and use classes that set their fields.

This lets mypy understand what fields are present, what their types are, etc. It's a little more work up-front, but the upside is that mypy is able to give you stronger guarantees about the correctness of your code.

If you find actually defining a bunch of classes w/ fields to be tedious, try using the new 'dataclasses' module (if you're using Python 3.7), or the 3rd party 'attrs' module. I believe mypy recently added support for both.

You might have to wait until mypy 0.620 is released this upcoming Tuesday if you want to use dataclasses though -- I don't remember if that feature made it in to mypy 0.600 or mypy 0.610.

Thanks for the explaination! I opted for a slightly different approach in the end. I replaced that class with one that sets it recursively and handles nested lists as well, but still opted with the 'Dict[str, Any]' approach like above in your answer. I then created a NamedTuple class which defined the types of the nested structures I actually wanted to use from the dict. That way the attributes can have their types set, keeping the flexibility I wanted in the first place for other parts of the code but without defeating the purpose of the static typing. — 264nm, Jul 06 '18 at 11:28

Type hints when representing dict keys by attribute notation in python 3.6

1 Answers1