Possible to break on module namespace assignments?

Question

I'm trying to track down the cause of a bug: https://github.com/numba/numba/issues/3027

It seems that for some older ubuntu installations, compiling a recursive numba function causes sys.stdout to be reset. I've verified that sys is not being reloaded, instead it must be that something somewhere is assigning to sys. So, I'd like to arrange for a breakpoint to be hit if members of the sys namespace are assigned to. Is this possible?

Reassigning: sys.__setattr__, __builtins__.setattr doesn't seem to catch it:

import sys

def f(*args, **kw):
    print >>sys.stderr, args, kw
    raise Exception("Break here")  # Never reached

print >>sys.stderr, sys.__setattr__
sys.__setattr__ = f
__builtins__.setattr = f
print >>sys.stderr, sys.__setattr__ #Changed


sys.stdout = 12345 # Should cause exception.

And also, there's something magic about the sys namespace:

pprint.pprint(dir(sys)) # No __setattr__ here
print "__setattr__" in dir(sys) # False. Nope

# But this:
print >>sys.stderr, sys.__setattr__  # <method-wrapper '__setattr__' of module object at 0x7f8065b6cbb0>

abarnert · Accepted Answer · 2018-06-20T23:33:45.517

__setattr__ is a special method:

For custom classes, implicit invocations of special methods are only guaranteed to work correctly if defined on an object’s type, not in the object’s instance dictionary.

Python doesn't document anywhere exactly which special methods skip the instance dictionary, and when they do so—in fact, it's really up to each implementation—but (at least in CPython and PyPy), __setattr__ always goes straight to the type.

Note that PEP 562 adds a special case for __getattr__ and __dir__ on ModuleType to Python 3.7, but does not include __setattr__.

So, assigning sys.__setattr__ has no effect.

This is also why '__setattr__' in dir(sys) is False—just as it is for anything that isn't a type. The dir function doesn't return attributes found on the class (or inherited from a base class). If you want to check for an attribute, you normally use hasattr(sys, '__setattr__')—or, better, just try to access the attribute (because that works even with, e.g., dynamic attributes created by custom __getattribute__).

Also, this means that the place to set your breakpoint would be types.ModuleType.__setattr__ (or type(sys).__setattr__, which is the same place). But that isn't going to work in CPython, because that's a C function slot on a builtin type (actually just inherited from object.__setattr__), not a Python method.

There are two traditional ways around this. Neither of them is guaranteed to work with builtin modules; from a quick test, with CPython 3.7, the first one works but the second doesn't. But try them both on your own Python implementation/version.

First, you can just create a subclass:

class HookedModuleType(types.ModuleType):
    def __setattr__(self, name, value):
        print(f'{self.__name__}.__setattr__({name}, {value})')
        return super().__setattr__(name, value)

… and then re-type the module:

mymodule.__class__ = HookedModuleType

Alternatively, since ModuleType doesn't override the default behavior of __setattr__, it just inherits it from object, which means that all it does is set self.__dict__[name] = value. So, you can write a dict that intercepts __setitem__ and get the same effect:

class HookedDict(dict):
    def __setitem__(self, key, value):
        print(f'{self._name}.__setitem__({key}, {value})')
        return super().__setitem__(key, value)

mymodule.__dict__ = HookedDict(mymodule.__dict__)
mymodule.__dict__._name = mymodule.__name__

If neither of these works, you have to create a slightly more complicated class that proxies to an actual module object:

class ModuleProxy(object):
    def __init__(self, module):
        object.__setattr__(self, '_module', module)
    def __getattr__(self, name):
        return getattr(self._module, name)
    def __delattr__(self, name):
        delattr(self._module, name)
    def __setattr__(self, name, value):
        print(f'{self._module.__name__}.__setattr__({name}, {value})')
        setattr(self._module, name, value)

… and then replace the module with a proxy to it:

sys = sys.modules['sys'] = ModuleProxy(sys)

This one is a bit easier to get wrong, and may cause some weird behavior in a few edge cases, but it doesn't rely on any non-guaranteed behavior, and it seems to work in CPython 3.7, 3.6, and 2.7 and PyPy 5.10/3.5 and 5.10/2.7 (obviously 2.7 requires changing the f-string to a format call).

Thanks. But `sys.__class__ = HookedModuleType` ==> `TypeError: __class__ assignment: only for heap types` :-( — user48956, Jun 20 '18 at 23:18
And `sys.__dict__ = HookedDict(sys.__dict__)` ==> `TypeError: readonly attribute` — user48956, Jun 20 '18 at 23:21
This did it. I didn't find a ModuleProxy class but @kindall shows how. — user48956, Jun 20 '18 at 23:26
@user48956 What do you mean you didn't find it? You should be able to just copy and paste it from my answer; you don't need to find it anywhere else. — abarnert, Jun 20 '18 at 23:27
Python 2.7 has no ModuleProxy class. @kindall's example does this nicely though. — user48956, Jun 20 '18 at 23:29
@user48956 Of course Python 2.7 has no ModuleProxy class. That's why I had to write one in my answer. [You can see it working here.](https://repl.it/repls/ParchedEmbellishedCensorware) — abarnert, Jun 20 '18 at 23:29
Cool... it showed up after reloading (or I missed it first time around). — user48956, Jun 20 '18 at 23:32
Thanks again -- this let me isolate and resolve a bug in numba. — user48956, Jun 21 '18 at 16:49

score 2 · Answer 2 · answered Jun 20 '18 at 23:16

A proxy or wrapper class will work:

class DebugModule(object):
    def __init__(self, module):
       self.__dict__["_module"] = module
    def __getattr__(self, name):
        return getattr(self._module, name)
    def __setattr__(self, name, value):
        setattr(self._module, name, value)
        print("{}.{} = {}".format(self._module.__name__, name, repr(value)))

Usage:

import sys
sys = DebugModule(sys)

Possible to break on module namespace assignments?

2 Answers2