Subclassing datatypes that have views in Python2.7 and Python3

Question

Python 3 introduced views (see this question). They were also backported to Python 2.7. I am currently in the process of subclassing dict in a Python 2.7 application (though with the goal to port it to Python 3 as well). I was wondering if - and how - I could subclass the .viewitems() and similar functions in such a way that they behave exactly like the original views.

Here is my intent: I have a dictonary like this:

data = my_subclassed_dict
data["_internal"] = "internal_value"
data["key"] = "value"
list(data.keys()) == ["key"]

That is, I filter everything stat starts with a "_". This works fine so far: For iterators I just yield and for lists I use a list comprehension that filters undesired values. However, those items have no connection the dict anymore (which is fine, it feels exactly like a dict). However, with both ways, this does not work:

keys = data.viewkeys()
"key" in keys
del data["key"]
"key" not in keys # This is False !

The last part does not work because there is no reference to the original keys, so Python won't notice.

So: Is there a simple way to achieve this (without re-implementing all of the logic!)?

This is more of a question out of interest as I don't see it to apply that much in my scenario.

Answer 1

The view objects basically 'empty' proxies. They point to the original dictionary.

Unfortunately, the current dictionary view objects are not really reusable. Quoting from the source code:

/* TODO(guido): The views objects are not complete:

 * support more set operations
 * support arbitrary mappings?
   - either these should be static or exported in dictobject.h
   - if public then they should probably be in builtins
*/

Note the support arbitrary mappings entry; these objects do not support arbitrary mappings, nor can we create new instances or subclass these from Python code:

>>> type({}.viewkeys())({})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: cannot create 'dict_keys' instances
>>> class MyView(type({}.viewkeys())): pass
... 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: Error when calling the metaclass bases
    type 'dict_keys' is not an acceptable base type

You are forced to create your own and implement all the hooks the view objects support:

class DictKeys(object):
    def __init__(self, parent):
        self.parent = parent

    def __len__(self):
        return len(self.parent)

    def __contains__(self, key):
        return key in self.parent

    def __iter__(self):
        return iter(self.parent)

etc.

The methods the original objects implement are:

>>> dir({}.viewkeys())
['__and__', '__class__', '__contains__', '__delattr__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__iter__', '__le__', '__len__', '__lt__', '__ne__', '__new__', '__or__', '__rand__', '__reduce__', '__reduce_ex__', '__repr__', '__ror__', '__rsub__', '__rxor__', '__setattr__', '__sizeof__', '__str__', '__sub__', '__subclasshook__', '__xor__']

The __and__, __or__, __sub__, __xor__, __rand__, __ror__, __rsub__ and __rxor__ methods implement overrides for the &, | and ^ operators to provide set operations.

If you are reasonably secure in reading C code, take a look at the view objects implementation to peek at how they implement their methods.

Answer 2

Consider storing any key that starts with an underscore in a separate dict. You won't have to implement keys, iterkeys, viewitems, etc, but you may need to reimplement a lot of other methods.