1

I'm trying to figure out what methods are called when a dict is unpacked, so that I can customize the process?

I suppose the following hack would have shown me the methods called during any method access of Dict instances:

class Dict(dict):
    def __getattribute__(self, name):
        print(name)
        return super().__getattribute__(name)

But the following session shows that no method is called to perform dict unpacking?

In [1]: d = Dict(a=1)
__class__
__class__

In [2]: {**d}
Out[2]: {'a': 1}

So what is actually happening here? How can I customize the unpacking process?

Edit

I don't think the question is a duplicate of the other one. Even implementing all the special methods mentioned in answers of that question, no of them are called during dict unpacking.

In [66]: class Dict(dict):
    ...:     def __getattribute__(self, name):
    ...:         print(name)
    ...:         return super().__getattribute__(name)
    ...:     def keys(self):
    ...:         print("hello")
    ...:         return super().keys()
    ...:     def __getitem__(self, key):
    ...:         print("hello")
    ...:         return super().__getitem__(key)
    ...:     def __len__(self):
    ...:         print("hello")
    ...:         return super().__len__()
    ...:     def __iter__(self):
    ...:         print("hello")
    ...:         return super().__iter__()
    ...:     

In [67]: d = Dict(a=1)
__class__
__class__

In [68]: {**d}
Out[68]: {'a': 1}

You can see no print line is called whatsoever. So my question remains unanswered.

FWIW, the python version is Python 3.6.5.

Naitree
  • 1,088
  • 1
  • 14
  • 23
  • Hello, @jonrsharpe does it still a duplicate? – Naitree Jun 28 '18 at 08:49
  • Don't subclass `dict`. – jonrsharpe Jun 28 '18 at 08:53
  • @jonrsharpe Thanks for that very enlightening remark. Are you trying to say because `dict` is a C-level builtin type, so that normal mapping methods access is bypassed and it looks like the internal memory is accessed directly? – Naitree Jun 28 '18 at 08:59
  • If "Don't subclass `dict`" is an answer, there will be no `collections.OrderedDict` or `django.http.request.QueryDict` for us to use, just to name two. – Naitree Jun 28 '18 at 09:10
  • 1
    @Naitree: As it happens, the bug you're hitting was fixed precisely because it (briefly) caused a problem with `collections.OrderedDict`. :-) – ShadowRanger Nov 08 '18 at 16:38

1 Answers1

1

This was a bug in how Python handled dict subclasses that was fixed in late September, 2018.

Before the fix, any dict subclass was converted to a normal dict using the concrete dict-specific C API methods (that bypass all dynamically defined overrides). After the fix, the code checks whether __iter__ (well, the C equivalent, tp_iter) has been overridden, and if so, it doesn't use the fast path for dicts. Checking __iter__ is slightly wrong IMO (the only two methods it actually uses are keys and __getitem__), but if you're overriding keys, you probably should be overriding __iter__ as well, so it's not a huge hassle to do so (in many cases, one can be an alias of the other, or at most a slim wrapper, depending on whether keys returns an iterator or a view object).

Given how recent the bugfix was, you'd need to upgrade Python to get the benefit. 3.6.7 and 3.7.1 are the first micro releases in their respective minor version lines that have the fix incorporated, so upgrading to either one should make your code work.

ShadowRanger
  • 143,180
  • 12
  • 188
  • 271
  • Just upgraded to CPython 3.7.1, it indeed works. The takeaway is you need to override `__iter__` in order to make subclass customization works. – Naitree Nov 09 '18 at 03:04