Altering traceback of a non-callable module

Question

I'm a minor contributor to a package where people are meant to do this (Foo.Bar.Bar is a class):

>>> from Foo.Bar import Bar
>>> s = Bar('a')

Sometimes people do this by mistake (Foo.Bar is a module):

>>> from Foo import Bar   
>>> s = Bar('a')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'module' object is not callable

This might seems simple, but users still fail to debug it, I would like to make it easier. I can't change the names of Foo or Bar but I would like to add a more informative traceback like:

TypeError("'module' object is not callable, perhaps you meant to call 'Bar.Bar()'")

I read the Callable modules Q&A, and I know that I can't add a __call__ method to a module (and I don't want to wrap the whole module in a class just for this). Anyway, I don't want the module to be callable, I just want a custom traceback. Is there a clean solution for Python 3.x and 2.7+?

I know you said you can't rename the modules, but that would really be the optimal solution. Renaming `Foo` to `foo` and `Bar` (the module) to `bar` would go a long way in indicating that they're not classes, and people would be less likely to confuse `bar` with `Bar`. — Aran-Fey, Jun 28 '18 at 15:45
@Aran-Fey Yes maybe but legacy is important here, a lot of code would break if I change the names, and I'm pretty sure that this would not be permitted — Chris_Rands, Jun 28 '18 at 15:47
You could rename it and provide a deprecated backwards compatible solution for a while. That way you could fix your tutorials and make new coffee more obvious without breaking old code without a warning. No idea about changing the message here though without the callable approach. — Voo, Jun 29 '18 at 11:42
Although I know this is not what you're asking for not and that it might not be helpful, I'm with @Aran-Fey and @Voo. The confusion comes with the naming of `Foo` and `Bar`, which should be in `snake_case` following PEP8. Trying to call a lowercase name to instantiate an object would immediately fall to your eye as a bit weird, at least for non-beginners in Python. — Jeronimo, Jul 13 '18 at 05:22

Mohsenasm · Answer 1 · 2018-07-17T19:54:56.377

4

Add this to top of Bar.py: (Based on this question)

import sys

this_module = sys.modules[__name__]
class MyModule(sys.modules[__name__].__class__):
    def __call__(self, *a, **k):  # module callable
        raise TypeError("'module' object is not callable, perhaps you meant to call 'Bar.Bar()'")
    def __getattribute__(self, name):
        return this_module.__getattribute__(name)

sys.modules[__name__] = MyModule(__name__)


# the rest of file
class Bar:
    pass

Note: Tested with python3.6 & python2.7.

edited Jul 17 '18 at 19:54

answered Jul 17 '18 at 19:11

Mohsenasm

2,916
1
18
22

This only works on Python 3.5 and up, because it relies on a [special case](https://github.com/python/cpython/blob/v3.5.0/Objects/typeobject.c#L3669) added in 3.5 that singles out modules as the only objects implemented in C that can have their `__class__` reassigned. – user2357112 Jul 17 '18 at 19:39
Now, this code can work for python2.7 too. :) @user2357112 – Mohsenasm Jul 17 '18 at 19:59
@user2357112 would you recommend a particular solution? – Chris_Rands Jul 17 '18 at 21:22
4

@Chris_Rands: I'd just live with the default error message. There are a bunch of ways you can change the error message, but every such way has subtle side effects and additional sources of confusion. For example, the (current) code in this answer breaks submodules if used in a package's `__init__.py`. Your current problems are easy to understand and easy to explain. The problems you'll give yourself by trying to change the error message are likely to be much harder to understand and explain. – user2357112 Jul 17 '18 at 21:36
@user2357112 Thank you, you are convincing me, I was hoping that would be a concise and clean solution, but I guess not. Can you say what problems might occur with the `excepthook` approach? – Chris_Rands Jul 18 '18 at 19:51
1

@Chris_Rands: The excepthook in Laurent LAPORTE's answer is extremely buggy, only examining local variables, and not attempting to check whether the `Bar` module was actually called as a function, rather than something else triggering a TypeError. Also, it's likely to interfere with other `excepthook` replacements, and it's incompatible with IPython. – user2357112 Jul 18 '18 at 20:02
@user2357112 Thank you! Yes I see, like with Laurent's answer `from Foo import Bar; 2 + ''` would raise the same (wrong) traceback and although fixable would be only the start of the issues. You've definitely convinced me. Although it is a not the answer I wanted, I think your comments alone are worthy of the bounty, it takes some authority to say this – Chris_Rands Jul 18 '18 at 21:46

Laurent LAPORTE · Answer 2 · 2018-07-17T12:58:08.123

What you want is to change the error message when is is displayed to the user. One way to do that is to define your own excepthook.

Your own function could:

search the calling frame in the traceback object (which contains informations about the TypeError exception and the function which does that),
search the Bar object in the local variables,
alter the error message if the object is a module instead of a class or function.

In Foo.__init__.py you can install a your excepthook

import inspect
import sys


def _install_foo_excepthook():
    _sys_excepthook = sys.excepthook

    def _foo_excepthook(exc_type, exc_value, exc_traceback):
        if exc_type is TypeError:
            # -- find the last frame (source of the exception)
            tb_frame = exc_traceback
            while tb_frame.tb_next is not None:
                tb_frame = tb_frame.tb_next

            # -- search 'Bar' in the local variable
            f_locals = tb_frame.tb_frame.f_locals
            if 'Bar' in f_locals:
                obj = f_locals['Bar']
                if inspect.ismodule(obj):
                    # -- change the error message
                    exc_value.args = ("'module' object is not callable, perhaps you meant to call 'Foo.Bar.Bar()'",)

        _sys_excepthook(exc_type, exc_value, exc_traceback)

    sys.excepthook = _foo_excepthook


_install_foo_excepthook()

Of course, you need to enforce this algorithm…

With the following demo:

# coding: utf-8

from Foo import Bar

s = Bar('a')

You get:

Traceback (most recent call last):
  File "/path/to/demo_bad.py", line 5, in <module>
    s = Bar('a')
TypeError: 'module' object is not callable, perhaps you meant to call 'Foo.Bar.Bar()'

Thanks, this makes sense, although feels a bit heavyweight for the underlying issue. Is it necessary to specify the encoding? — Chris_Rands, Jul 17 '18 at 08:17
@Chris_Rands: no, it is a copy/paste mistake. On Python 2.7 it can be useful but Python Exception doesn't support unicode string very well: choose messages in ASCII. — Laurent LAPORTE, Jul 17 '18 at 13:00

user2357112 · Accepted Answer · 2018-07-18T22:34:32.077

There are a lot of ways you could get a different error message, but they all have weird caveats and side effects.

Replacing the module's __class__ with a types.ModuleType subclass is probably the cleanest option, but it only works on Python 3.5+.

Besides the 3.5+ limitation, the primary weird side effects I've thought of for this option are that the module will be reported callable by the callable function, and that reloading the module will replace its class again unless you're careful to avoid such double-replacement.
Replacing the module object with a different object works on pre-3.5 Python versions, but it's very tricky to get completely right.

Submodules, reloading, global variables, any module functionality besides the custom error message... all of those are likely to break if you miss some subtle aspect of the implementation. Also, the module will be reported callable by callable, just like with the __class__ replacement.
Trying to modify the exception message after the exception is raised, for example in sys.excepthook, is possible, but there isn't a good way to tell that any particular TypeError came from trying to call your module as a function.

Probably the best you could do would be to check for a TypeError with a 'module' object is not callable message in a namespace where it looks plausible that your module would have been called - for example, if the Bar name is bound to the Foo.Bar module in either the frame's locals or globals - but that's still going to have plenty of false negatives and false positives. Also, sys.excepthook replacement isn't compatible with IPython, and whatever mechanism you use would probably conflict with something.

Right now, the problems you have are easy to understand and easy to explain. The problems you would have with any attempt to change the error message are likely to be much harder to understand and harder to explain. It's probably not a worthwhile tradeoff.

Altering traceback of a non-callable module

3 Answers3