Why don't I see errors from readline.set_completion_display_matches_hook?

Question

Consider this code:

#!/usr/bin/env python3

from cmd import Cmd
import readline

class mycmd(Cmd):
    def match_display_hook(self, substitution, matches, longest_match_length):
        someNonexistentMethod()
        print()
        for match in matches:
            print(match)
        print(self.prompt, readline.get_line_buffer(), sep='', end='', flush=True)

    def do_crash(self, s):
        someNonexistentMethod()

    def do_quit(self, s):
        return True

if __name__ == '__main__':
    obj = mycmd()
    readline.set_completion_display_matches_hook(obj.match_display_hook)
    obj.cmdloop()

I expect to see NameError: name 'someNonexistentMethod' is not defined when I run that and hit TabTab. However, nothing actually seems to happen at all (the error does occur, so the other functions that would print the completion don't run; I just don't see the error). I do see the expected error when I run crash, so I know error handling works fine in the program overall, but is just broken inside of the set_completion_display_matches_hook callback. Why is this, and can I do something about it?

BStadlbauer · Answer 1 · 2020-11-15T19:24:00.987

TL;DR

It looks as if readline C-Binding just ignores exceptions when calling the hook, when TabTab is pressed.

I believe the root of the problem might be these lines (1033-1049) in the C-binding readline.c

    r = PyObject_CallFunction(readlinestate_global->completion_display_matches_hook,
                              "NNi", sub, m, max_length);

    m=NULL;

    if (r == NULL ||
        (r != Py_None && PyLong_AsLong(r) == -1 && PyErr_Occurred())) {
        goto error;
    }
    Py_CLEAR(r);

    if (0) {
    error:
        PyErr_Clear();
        Py_XDECREF(m);
        Py_XDECREF(r);
    }

Where, if an error occurs, it is just cleared. Reference to PyErr_Clear()

Steps I used for debugging:

Check if the Exception is raised

I changed the function to:

def match_display_hook(self, substitution, matches, longest_match_length):
    try:
        someNonexistentMethod()
    except Exception as e:
        print(e)

which then printed name 'someNonexistentMethod' is not defined as expected (and all the other, expected, outputs). Raising any other Exception here did not quit the command prompt.

Check if printing to `stderr` works

At last, I checked if one could print to sys.stderr by adding:

def match_display_hook(self, substitution, matches, longest_match_length):
    print("foobar", file=sys.stderr, flush=True)

which printed foobar as expected.

traal · Accepted Answer · 2020-11-17T13:05:03.527

Why?

I would guess that this is by design. According to rlcompleter docs:

Any exception raised during the evaluation of the expression is caught, silenced and None is returned.

See the rlcompleter source code for the rationale:

Exceptions raised by the completer function are ignored (and generally cause the completion to fail). This is a feature -- since readline sets the tty device in raw (or cbreak) mode, printing a traceback wouldn't work well without some complicated hoopla to save, reset and restore the tty state.

Workaround

As a workaround, for debugging, wrap your hook in a function that catches all exceptions (or write a function decorator), and use the logging module to log your stack traces to a file:

import logging
logging.basicConfig(filename="example.log", format='%(asctime)s %(message)s')

def broken_function():
    raise NameError("Hi, my name is Name Error")

def logging_wrapper(*args, **kwargs):
    result = None
    try:
        result = broken_function(*args, **kwargs)
    except Exception as ex:
        logging.exception(ex)
    return result

logging_wrapper()

This script runs successfully, and example.log contains both the log message and the stack trace:

2020-11-17 13:55:51,714 Hi, my name is Name Error
Traceback (most recent call last):
  File "/Users/traal/python/./stacktrace.py", line 12, in logging_wrapper
    result = function_to_run()
  File "/Users/traal/python/./stacktrace.py", line 7, in broken_function
    raise NameError("Hi, my name is Name Error")
NameError: Hi, my name is Name Error