Can obj.__dict__ be inspected if the class has a __dict__ variable?

Question

I am interested in whether there is a way to introspect a Python instance infallibly to see its __dict__ despite any obstacles that the programmer might have thrown in the way, because that would help me debug problems like unintended reference loops and dangling resources like open files.

A simpler example is: how can I see the keys of a dict subclass if the programmer has hidden keys() behind a class of its own? The way around that is to manually call the dict keys() method instead of letting inheritance call the subclass's version of the method:

# Simple example of getting to the real info
# about an instance

class KeyHidingDict(dict):
    def keys(self):
        return []  # there are no keys here!

khd = KeyHidingDict(a=1, b=2, c=3)
khd.keys()       # drat, returns []
dict.keys(khd)   # aha! returns ['a', 'b', 'c']

Now my actual question is, how can I see the __dict__ of an instance, no matter what the programmer might have done to hide it from me? If they set a __dict__ class variable then it seems to shadow the actual __dict__ of any objects inherited from that class:

# My actual question

class DunderDictHider(object):
    __dict__ = {'fake': 'dict'}

ddh = DunderDictHider()
ddh.a = 1
ddh.b = 2
print ddh.a        # prints out 1
print ddh.__dict__ # drat, prints {'fake': 'dict'}

This false value for __dict__ does not, as you can see, interfere with actual attribute setting and getting, but it does mislead dir() by hiding a and b and displaying fake as the object's instance variable instead.

Again, my goal is to write a tool that helps me introspect class instances to see “what is really going on” when I am wondering why a set of class instances is taking so much memory or holding so many files open — and even though the situation above is extremely contrived, finding a way around it would let the tool work all the time instead of saying “works great, unless the class you are looking at has… [description of the exceptional situation above].”

I had thought I would be able to infallibly grab the __dict__ with something like:

dict_descr = object.__dict__['__dict__']
print dict_descr(ddh, DunderDictHider)

But it turns out that object does not have a __dict__ descriptor. Instead, the subtype_dict() C function seems to get separately attached to each subclass of object that the programmer creates; there is no central way to name or fetch the descriptor so that it can be manually applied to objects whose class shadows it.

Any ideas, anyone? :)

Answer 1

I'm not sure I'm happy with how simple this is:

>>> class DunderDictHider(object):
...     __dict__ = {'fake': 'dict'}
... 
>>> ddh = DunderDictHider()
>>> ddh.a = 1
>>> ddh.b = 2
>>> 
>>> print ddh.a
1
>>> print ddh.__dict__
{'fake': 'dict'}

The problem is that the class is cheating? Fix that!

>>> class DictUnhider(object):
...     pass
... 
>>> ddh.__class__ = DictUnhider
>>> print ddh.a
1
>>> print ddh.__dict__
{'a': 1, 'b': 2}

And there it is. This completely fails though, if the class defines any slots.

>>> class DoesntHaveDict(object):
...     __slots__ = ['a', 'b']
... 
>>> dhd = DoesntHaveDict()
>>> dhd.a = 1
>>> dhd.b = 2
>>> 
>>> print dhd.a
1
>>> dhd.__class__ = DictUnhider
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: __class__ assignment: 'DoesntHaveDict' object layout differs from 'DictUnhider'
>>> 

Answer 2

This one is based on Jerub answer in this topic: What is a metaclass in Python?

You can achieve what you're looking for with metaclasses.

First you need to create a metaclass:

def test_metaclass(name, bases, dict):
    print 'The Class Name is', name
    print 'The Class Bases are', bases
    print 'The dict has', len(dict), 'elems, the keys are', dict.keys()

    return dict

of course the prints are not necessery.

Then let me introduce your new DunderDictHider:

class DunderDictHider(object):
    __metaclass__ = test_metaclass
    __dict__ = {'fake': 'dict'}

Now you have access to all initialized elems by repr(DunderDictHider)

The output (with print repr(DunderDictHider) line):

The Class Name is DunderDictHider
The Class Bases are (<type 'object'>,)
The dict has 3 elems, the keys are ['__dict__', '__module__', '__metaclass__']
{'__dict__': {'fake': 'dict'}, '__module__': '__main__', '__metaclass__': <function test_metaclass at 0x1001df758>}

Each time you can try

if '__dict__' in repr(DunderDictHider)

to know whether this class tries to hide its __dict__ or not. Remember, that repr output is a string. It can be done better, but that's the idea itself.