iPython debugger raises `NameError: name ... is not defined`

Question

I can't understand the following exception that is raised in this Python debugger session:

(Pdb) p [move for move in move_values if move[0] == max_value]
*** NameError: name 'max_value' is not defined
(Pdb) [move for move in move_values]
[(0.5, (0, 0)), (0.5, (0, 1)), (0.5, (0, 2)), (0.5, (1, 0)), (0.5, (1, 1)), (0.5, (1, 2)), (0.5, (2, 0)), (0.5, (2, 1)), (0.5, (2, 2))]
(Pdb) max_value
0.5
(Pdb) (0.5, (0, 2))[0] == max_value
True
(Pdb) [move for move in move_values if move[0] == 0.5]
[(0.5, (0, 0)), (0.5, (0, 1)), (0.5, (0, 2)), (0.5, (1, 0)), (0.5, (1, 1)), (0.5, (1, 2)), (0.5, (2, 0)), (0.5, (2, 1)), (0.5, (2, 2))]
(Pdb) [move for move in move_values if move[0] == max_value]
*** NameError: name 'max_value' is not defined

Why is it sometimes telling me max_value is not defined and other times not?

Incidentally, this is the code immediately prior to the debugger starting:

max_value = max(move_values)[0]
best_moves = [move for move in move_values if move[0] == max_value]
import pdb; pdb.set_trace()

I am using Python 3.6 running in PyCharm.

AMENDED UPDATE:

After more testing it appears that local variables are not visible within list comprehensions within a pdb session when I do the following from an iPython REPL or in PyCharm:

$ ipython
Python 3.6.5 | packaged by conda-forge | (default, Apr  6 2018, 13:44:09) 
Type 'copyright', 'credits' or 'license' for more information
IPython 6.4.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import pdb; pdb.set_trace()
--Call--
> /Users/billtubbs/anaconda/envs/py36/lib/python3.6/site-packages/IPython/core/displayhook.py(247)__call__()
-> def __call__(self, result=None):
(Pdb) x = 1; [x for i in range(3)]
*** NameError: name 'x' is not defined

But in a regular Python REPL it works:

$ python
Python 3.6.5 | packaged by conda-forge | (default, Apr  6 2018, 13:44:09) 
[GCC 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.53)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pdb; pdb.set_trace()
--Return--
> <stdin>(1)<module>()->None
(Pdb) x = 1; [x for i in range(3)]
[1, 1, 1]

I tested above with versions 3.4, 3.5, 3.6 so it does not appear to be version dependent.

UPDATE 2

Please note, the above test ('AMENDED UPDATE') is problematic because it uses import pdb; pdb.set_trace() in the interactive REPL.

Also, the original problem is not limited to iPython.

See answer by user2357112 below for a comprehensive explanation of what is going on here.

Sorry if I caused any confusion!

Answer 1

A possible solution/workaround is to run

 globals().update(locals())

before running the list comprehension in (i)pdb.

Answer 2

You've got two core problems here. The first is that (when calling pdb.set_trace() interactively in IPython) you're debugging IPython's guts instead of the scope you wanted. The second is that list comprehension scoping rules interact badly with cases where the variables present in enclosing scopes can't be determined statically, such as in debuggers or class bodies.

The first problem pretty much only happens when typing pdb.set_trace() into an IPython interactive prompt, which isn't a very useful thing to do, so the simplest way to avoid the problem is to just not do that. If you want to do it anyway, you can enter the r command a few times until pdb says you're out of IPython's guts. (Don't overshoot, or you'll end up in a different part of IPython's guts.)

The second problem is an essentially unavoidable interaction of heavily entrenched language design decisions. Unfortunately, it's unlikely to go away. List comprehensions in a debugger only work in a global scope, not while debugging a function. If you want to build a list while debugging a function, the easiest way is probably to use the interact command and write a for loop.

---

Here's the full combination of effects going into this.

1. pdb.set_trace() triggers pdb on the next trace event, not at the point where pdb.set_trace() is called.

The trace function mechanism used by pdb and other Python debuggers only triggers on certain specific events, and "when a trace function is set" is unfortunately not one of those events. Normally, the next event is either a 'line' event for the next line or a 'return' event for the end of the current code object's execution, but that's not what happens here.

2. IPython sets a displayhook to customize expression statement handling.

The mechanism Python uses to display the result of expression statements is sys.displayhook. When you do 1+2 at the interactive prompt:

>>> 1+2
3

sys.displayhook is what prints the 3 instead of discarding it. It also sets _. When the result of an expression statement is None, such as with the expression pdb.set_trace(), sys.displayhook does nothing, but it's still called.

IPython replaces sys.displayhook with its own custom handler, responsible for printing the Out[n]: thingy, for setting entries in the Out record, for calling IPython custom pretty-printing, and for all sorts of other IPython conveniences. For our purposes, the important thing is that IPython's displayhook is written in Python, so the next trace event is a 'call' event for the displayhook.

pdb starts debugging inside IPython's displayhook.

In [1]: import pdb; pdb.set_trace()
--Call--
> /Users/billtubbs/anaconda/envs/py36/lib/python3.6/site-packages/IPython/core/displayhook.py(247)__call__()
-> def __call__(self, result=None):

3. List comprehensions create a new scope.

People didn't like how list comprehensions leaked the loop variable into the containing scope in Python 2, so list comprehensions get their own scope in Python 3.

4. pdb uses eval, which interacts really poorly with closure variables.

Python's closure variable mechanism relies on static scope analysis that's completely incompatible with how eval works. Thus, new scopes created inside eval have no access to closure variables; they can only access globals.

---

Putting it all together, in IPython, you end up debugging the IPython displayhook instead of the scope you're running interactive code in. Since you're inside IPython's displayhook, your x = 1 assignment goes into the displayhook's locals. The subsequent list comprehension would need access to the displayhook's locals to access x, but that would be a closure variable to the list comprehension, which doesn't work with eval.

Outside of IPython, sys.displayhook is written in C, so pdb can't enter it, and there's no 'call' event for it. You end up debugging the scope you intended to debug. Since you're in a global scope, x = 1 goes in globals, and the list comprehension can access it.

You would have seen the same effect if you tried to run x = 1; [x for i in range(3)] while debugging any ordinary function.

Answer 3

Inspired by Kobi T's answer, if you want the workaround to apply without manual intervention every time this problem comes up, you can subclass TerminalPdb (ipython's debugger, but it should also work by subclassing Pdb). The key is extending TerminalPdb's (really, Pdb's, as it's inherited) default method. The default method is the one called whenever you run a line of code from Pdb's debugger prompt.

As user2357112-supports-monica's answer described well, this function uses exec to run the code, and typically captures any errors raised and prints them out instead of raising them. You can make use of that fact that for the particular problem of list comprehensions (and closures more generally) that appears here, a NameError gets raised. Specifically, if that's the raised error you can intercede to apply Kobi T's workaround (a version of globals().update(locals())), and then exec the code again--if the NameError was raised because of the closure problem described here, the code should now run with no errors; if not, the NameError will get raised again and the output will be as expected for that error.

An added benefit here relates to how globals are handled. For the purposes of the code run by exec, the global/local variables it sees can be explicitly defined as inputs to exec. As a result, instead of updating the workspace's globals(), you can update just the copy that gets sent to exec, keeping your workspace a little less affected when you want to run these list comprehensions.

import sys
import traceback
from IPython.terminal.debugger import TerminalPdb

class LcTerminalPdb(TerminalPdb):
    # the default method is what runs code written to the prompt of the debugger
    def default(self, line):
        # most of this method directly copies the original one, but there's no
        # good way to add the NameError handling separately from the original
        # code
        if line[:1] == '!': line = line[1:]
        locals = self.curframe_locals
        globals = self.curframe.f_globals
        try:
            code = compile(line + '\n', '<stdin>', 'single')
            save_stdout = sys.stdout
            save_stdin = sys.stdin
            save_displayhook = sys.displayhook
            try:
                sys.stdin = self.stdin
                sys.stdout = self.stdout
                sys.displayhook = self.displayhook
                exec(code, globals, locals)
            ''' BELOW IS THE CODE ADDED TO Pdb's default()'''
            except NameError:
                # NameError occurs when a list comprehension requires variables
                # to be bound in its closure, but isn't able to because of how
                # exec handles local variables; putting the variable in the
                # global dictionary works, and this code takes the sledgehammer
                # approach of assigning *all* locals to globals, so we don't
                # have to be picky about which variable exactly was needed
                try:
                    tempGlobal = globals.copy()
                    tempGlobal.update(locals)
                    exec(code, tempGlobal, locals)
                except:
                    raise
            '''ABOVE IS THE CODE ADDED TO Pdb's default()'''
            finally:
                sys.stdout = save_stdout
                sys.stdin = save_stdin
                sys.displayhook = save_displayhook
        except:
            self._error_exc()

    # TerminalPdb doesn't directly call _error_exc, which was originally defined
    # in Pdb, so we can't import it from there, and it's underscored, so it
    # isn't immediately accessible from TerminalPdb's parent Pdb. However, it's
    # a simple function so I'm just replicating it here.
    def _error_exc(self):
        exc_info = sys.exc_info()[:2]
        self.error(traceback.format_exception_only(*exc_info)[-1].strip())

Here's the result we want:

>> import pdb; pdb.set_trace()
(Pdb) x=5;[x for i in range(3)]
*** NameError: name 'x' is not defined
(Pdb) q
>> import LcTerminalPdb; LcTerminalPdb.set_trace()
ipdb> x=5;[x for i in range(3)]
[5, 5, 5] 

This solution will work if you call LcTerminalPdb.set_trace(). Alternatively, you can save this as a module on the Python path, in that module define a set_trace method that calls LcTerminalPdb.set_trace() (below) and then set the PYTHONBREAKPOINT environment variable to point to it.

[appended to file with class code from above]
def set_trace(frame=None):
    """
    Start debugging from `frame`.
    If frame is not specified, debugging starts from caller's frame.
    """
    LcTerminalPdb().set_trace(frame or sys._getframe().f_back)

I made a quick LcTerminalPdb git repo with this file and instructions on how to get that part working for anyone interested.