Detect changes to a nested dictionary with Python

Question

I've seen a few similar questions on SO regarding detecting changes to a dictionary and calling a function when the dictionary changes, such as:

These examples use variations of the Observer pattern or overloading __setitem__, but all these examples don't detect changes on nested dictionary values.

For example, if I have:

my_dict = {'a': {'b': 1}}
my_dict['a']['b'] = 2

The assignment of 2 to the element ['a']['b'] will not be detected.

I'm wondering if there is an elegant way of detecting changes not only to the base elements of a dictionary but all the child elements of a nested dictionary as well.

What does it mean to detect changes? How are the changes being made without your knowledge? I assume there are some broader aspects that you haven't touched on here. — roganjosh, Dec 21 '18 at 09:33
`my_dict['a']...` is getting, not setting, an item, then `...['b'] = 2` is setting on the child. The child also has to have the custom behaviour. — jonrsharpe, Dec 21 '18 at 09:36
If you have a look at the example links provided, you will see what I mean. I think it's pretty clear. For example, when `my_dict['a']['b'] = 2` is assigned, I would like to be able to detect that the value `['a']['b']` in the nested dictionary has changed and run a specific function based on what has changed. — Willem van Ketwich, Dec 21 '18 at 09:36
Well, at the moment, you just give an example using a regular dictionary. Presumably you have subclassed it — roganjosh, Dec 21 '18 at 09:39
As given in the second link, why don't we recursively convert all nested dicts to custom ``MyDict``s subclassing ``dict``? — Deepak Saini, Dec 21 '18 at 09:43
roganjosh, I think you're just splitting hairs over what I have described. if you take it in the context of the example links given, it's pretty clear. — Willem van Ketwich, Dec 21 '18 at 10:03

ma3oun · Accepted Answer · 2018-12-21T10:54:25.710

7

Building on the answer given in here, just do the following:

class MyDict(dict):
    def __setitem__(self, item, value):
        print("You are changing the value of {} to {}!!".format(item, value))
        super(MyDict, self).__setitem__(item, value)

and then:

my_dict = MyDict({'a': MyDict({'b': 1})})

my_dict['a']['b'] = 2

You are changing the value of b to 2!!

my_dict['a'] = 5

You are changing the value of a to 5!!

If you want to avoid manual calls to MyDict at each nesting level, one way of doing it, is to fully overload the dict class. For example:

class MyDict(dict):
    def __init__(self,initialDict):
        for k,v in initialDict.items():
          if isinstance(v,dict):
            initialDict[k] = MyDict(v)
        super().__init__(initialDict)

    def __setitem__(self, item, value):
        if isinstance(value,dict):
          _value = MyDict(value)
        else:
          _value = value
        print("You are changing the value of {} to {}!!".format(item, _value))
        super().__setitem__(item, _value)

You can then do the following:

# Simple initialization using a normal dict synthax
my_dict = MyDict({'a': {'b': 1}})

# update example
my_dict['c'] = {'d':{'e':4}}

You are changing the value of c to {'d': {'e': 4}}!!

my_dict['a']['b'] = 2
my_dict['c']['d']['e'] = 6

You are changing the value of b to 2!!

You are changing the value of e to 6!!

edited Dec 21 '18 at 10:54

answered Dec 21 '18 at 09:45

ma3oun

3,681
1
21
33

2

Nice one, maybe it would be interesting to convert directly the nested dicts to MyDict objects (using `isinstance` on the `value`) – Corentin Limier Dec 21 '18 at 09:48
thanks for your answer, ma3oun - this is one way of doing it, however what Corentin Limier describes would be a more 'elegant' solution imho. – Willem van Ketwich Dec 21 '18 at 10:00
1

See the edit: Solution without modifying the built dictionary. However, you would need to add the other `dict` constructors for a complete solution including the use of kwargs – ma3oun Dec 21 '18 at 10:55
thanks, this is the best solution so far. it would be interesting to see if we could identify the full context of the change. e.g `You are changing the value of ['c']['d']['e'] to 6!!`. – Willem van Ketwich Dec 21 '18 at 11:10
1

It can be done easily by adding a `parent` attribute in the constructor. You initialize it to `None` for the root dict and fill it with the `key` when setting a child dict item (you also need to set it in `__init__` for all dict values. – ma3oun Dec 21 '18 at 11:50

Deepak Saini · Answer 2 · 2018-12-21T10:17:22.887

Complete solution borrowing from the this link(the second one given by OP)

class MyDict(dict):
    def __setitem__(self, item, value):
        print("You are changing the value of {key} to {value}!!".format(key=item, value=value))
        super(MyDict, self).__setitem__(item, convert_to_MyDict_nested(value))

def convert_to_MyDict_nested(d):
    if not(isinstance(d, dict)):
        return d
    for k, v in d.items():
        if isinstance(v, dict):
            d[k] = convert_to_MyDict_nested(v)
    return MyDict(d)

So that if

d = {'a': {'b': 1}}

then,

d = convert_to_MyDict_nested(d)
d['a']['b'] = 2  # prints You are changing the value of b to 2!!
d['a']= 5  # prints You are changing the value of a to 5!!

Also, edited according to comment by OP. So,

d["c"] = {"e" : 7}  # prints You are changing the value of c to {'e': 7}!!
d["c"]["e"] = 9  # prints You are changing the value of e to 9!!

Is there a way to do this internally to the class `MyDict`? The reason I ask is if we set `d['a'] = {'b': 6}` then do `d['a']['b'] = 7`, the second assignment will not be captured as it isn't of type `MyDict`. — Willem van Ketwich, Dec 21 '18 at 10:12
yeah, we change to ``super(MyDict, self).__setitem__(item, convert_to_MyDict_nested(value))`` so that, all dicts ever put inside the initial MyDict and MyDicts. — Deepak Saini, Dec 21 '18 at 10:16

Lie Ryan · Answer 3 · 2018-12-21T10:30:04.487

1

You can write an adapter class that automatically wraps values in itself, like this (untested, but I think it illustrates the point):

class DictChangeListenerAdapter:
    def __init__(self, original, listener):
        self._original = original
        self._listener = listener

    def __getitem__(self, key):
        value = self._original.__getitem__(key)
        if isinstance(value, dict):
            return DictChangeListenerAdapter(self._original[key], self._listener)
        else:
            return value

    def __setitem__(self, key, value):
        self._original.__setitem__(key, value)
        self._listener(self, key, value)

Note that this is going to cause access to the wrapped items to be much more expensive, use with care.

edited Dec 21 '18 at 10:30

answered Dec 21 '18 at 10:24

Lie Ryan

62,238
13
100
144

thanks for this, although printing a parent dict doesn't give a dict but the adapter object.. ? `<__main__.DictChangeListenerAdapter object at 0x10dd867f0>` – Willem van Ketwich Dec 21 '18 at 10:45
1

You'd need to unwrap it first before printing. Or you can implement `__repr__` and forward it to the underlying dict. – Lie Ryan Dec 21 '18 at 10:51

Detect changes to a nested dictionary with Python

3 Answers3