Custom dictionary to maintain it's getitem on ** (star-star-unpacking)

Question

Merry christmas everybody,

I'm implementing a custom dictionary that allows attribute access, e.g. dct.attribute. The dictionaries can be nested, so dct.nested_dct.attribute should also be possible. This is working pretty well already, except for the star-star-unpacking. I think I'm able to express what I'm trying to do better using code than words. So here is the class I'm writing. The tests should explain pretty clearly what it does:

class DotDict(dict):
    def __getattr__(self, item):
        return self.__getitem__(item)

    def __getitem__(self, item):
        item = super().__getitem__(item)
        if isinstance(item, dict):
            return self.__class__(item)
        return item


class TestDotDict:
    @pytest.fixture
    def dot_dict(self):
        input_dict = dict(
            a=1,
            b=dict(
                c=2,
                d=3,
            )
        )
        return DotDict(input_dict)

    def test_can_access_by_dot(self, dot_dict):
        assert dot_dict.a == 1

    def test_returned_dicts_are_dot_dicts(self, dot_dict):
        b_dict = dot_dict["b"]
        assert isinstance(b_dict, DotDict)
        assert b_dict.c == 2

    def test_getting_item_also_returns_dot_dicts(self, dot_dict):
        b_dict = dot_dict["b"]
        assert isinstance(b_dict, DotDict)
        assert b_dict.c == 2

    def test_unpack_as_function_arguments_yields_dot_dicts_for_children(self, dot_dict):
        # this is failing
        def checker(a, b):
            assert a == 1
            assert b.c == 2
        checker(**dot_dict)

As stated in the comment, the last test is failing. Does anybody know how to fix it?

Following the answers from this question: star unpacking for own classes , I figured I need to inherit from collections.abc.Mapping and dict. However, this didn't solve the issue.

I was thinking this might be related to a MRO that's not entirely clear to me. But no matter if I change the class definition to

class DotDict(Mapping, item):

or

class DotDict(item, Mapping):

my tests won't become green.

score 1 · Answer 1 · answered Dec 24 '18 at 14:54

In test_star_star_mapping_maintains_child_dot_dicts you are creating a dict not a DotDict so, refactoring to:

def test_star_star_mapping_maintains_child_dot_dicts(self, dot_dict):
    obtained_via_star = DotDict(dict(**dot_dict))
    b_dict = obtained_via_star["b"]
    assert b_dict.c == 2

Will make the test pass because you now you are creating a DotDict. Maybe you want to remove the part dict(**dot_dict) so this version also works:

def test_star_star_mapping_maintains_child_dot_dicts(self, dot_dict):
    obtained_via_star = DotDict(**dot_dict)
    b_dict = obtained_via_star["b"]
    assert b_dict.c == 2

Unfortunately not. So of course it get's the tests green. But I want to do be able to do the unpacking return `DotDict` to pass it dynamically along as function arguments (like `**kwargs`). Therefore wrapping it in a `DotDict` again doesn't work. I edited the question and the tests to make it clearer. — lmr2391, Dec 24 '18 at 15:15

score 1 · Accepted Answer · answered Dec 24 '18 at 15:35

The problem you are facing is that yu are trying to build upon a native dict - and for this class, __getitem__ is just one of several ways its values can be retrieved. Due to the way dicts are implemented in Python, for both historic and performance reasons, there are lots of ways that will simply bypass __getitem__ altogether, and therefore, nested dictionaries will never be "wrapped" in a DotDict. (for example: .values(), items(), and starmap will possibly bypass even these)

What you really want there is to subclass collections.abc.MutableMapping - it is constructed in a way that ensure that any item retrieval will go through __getitem__, (you will have to implement the methods indicated in the documentation though, including __delitem__, __setitem__ and __iter__ - the recomendation is to keep the actual data as a plain dictioanry in a .data attribute created in the __init__ method).

Perceive that this also gives you better control on your data, enabling you, for example, to wrap your data in your custom-class directly on setitem, and jsut don't care on attribute retrieval - or, the other way around, store any mappings as plain dictionaries for memory-saving and efficiency and wrap it on retrieval.

Great, thanks for the answer. Do you know what's the difference between `Mapping` and `MutableMapping`? The documentation is a bit short on this. — lmr2391, Dec 24 '18 at 15:47

score 0 · Answer 3 · answered Dec 24 '18 at 15:16

0

wow, try to run the following code with uncommented __iter__

class DotDict(dict):
#    def __iter__(self):
#        return super().__iter__()

    def __getattr__(self, item):
        return self.__getitem__(item)

    def __getitem__(self, item):
        item = super().__getitem__(item)
        if isinstance(item, dict):
            return self.__class__(item)
        return item

d = DotDict({'a': {'b':'c'}})

print(type(dict(**d)['a']))

very, very strange

answered Dec 24 '18 at 15:16

Paweł Kordowski

2,688
1
14
21

Didn't help for me. Does it for you? – lmr2391 Dec 24 '18 at 15:22
with `__iter__` implemented I get ``, without `` – Paweł Kordowski Dec 24 '18 at 15:24
What python are you on? I'm using 3.6 – lmr2391 Dec 24 '18 at 15:28
Apparently the `__iter__` is not called at all – lmr2391 Dec 24 '18 at 15:28
I use Python 3.6.7 – Paweł Kordowski Dec 24 '18 at 15:29

Custom dictionary to maintain it's __getitem__ on ** (star-star-unpacking)

3 Answers3

Custom dictionary to maintain it's getitem on ** (star-star-unpacking)