3

I hit a scoping issue today that initially surprised me. It can be readily demonstrated by the following:

def scope():
    x = 1
    def working():
        print x
    def broken():
        import pdb; pdb.set_trace()
    working()
    broken()


Python 2.7.12 (default, Jul  1 2016, 15:12:24)
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
...
>>> scope()
1
--Return--
> <stdin>(6)broken()->None
(Pdb) x
*** NameError: name 'x' is not defined

Python 3.5.2 (default, Sep 10 2016, 08:21:44)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> def scope():
...
>>> scope()
1
--Return--
> <stdin>(6)broken()->None
(Pdb) x
*** NameError: name 'x' is not defined
(Pdb)

So it appears a scope will only contain outer-scope values if there's an explicit reference to them at compilation time. This is certainly what appears to happen when looking at the bytecode:

  6           0 LOAD_GLOBAL              0 (x)
              3 PRINT_ITEM          
              4 PRINT_NEWLINE       
              5 LOAD_CONST               0 (None)
              8 RETURN_VALUE

Is this an artifact of how Python differentiates between outer & inner scope references by examining the function for bindings?

This does not appear to be explained by the Python scoping rules:

Although scopes are determined statically, they are used dynamically. At any time during execution, there are at least three nested scopes whose namespaces are directly accessible:

  • the innermost scope, which is searched first, contains the local names
  • the scopes of any enclosing functions, which are searched starting with the nearest enclosing scope, contains non-local, but also non-global names

What I'm trying to understand is the conflict between the two highlighted parts above: scopes are static but the promise of what names are accessible doesn't seem to be upheld.

Specifically, what I'm looking for is official documentation of this explicit behaviour.

Community
  • 1
  • 1
Matthew Trevor
  • 14,354
  • 6
  • 37
  • 50
  • 1
    An inner scope doesn't contain an outer-scope object, but it can read it if it's not shadowed by an object with the same name in an inner scope. For details, please see [Python Scopes and Namespaces](https://docs.python.org/2/tutorial/classes.html#python-scopes-and-namespaces). – PM 2Ring Nov 22 '16 at 07:20
  • Yes, binding (assignment) in a function makes a variable local to that function, if there's no `global` or `nonlocal` statement declaring it otherwise. This is indeed done statically at compile time. In `working()` Python knows `x` is not local so loads it with the `LOAD_GLOBAL` opcode (which you can see by disassembling with the `dis` module). – kindall Nov 22 '16 at 07:43
  • @PM2Ring I noticed this when I set a `pdb` breakpoint in the inner function and tried to inspect the outer scope. I wasn't shadowing any object but still couldn't "read" outer scope objects. It's the necessity for the name to be present at compile time that is confusing, this isn't a shadowing issue. – Matthew Trevor Nov 22 '16 at 08:19
  • @kindall Cheers but i'd like to confirm if this is documented behaviour or an artifact of CPython's implementation. – Matthew Trevor Nov 22 '16 at 08:20
  • It's documented. https://docs.python.org/3.3/reference/executionmodel.html - ”If a name is bound in a block, it is a local variable of that block” – kindall Nov 22 '16 at 08:28
  • @kindall But the same page says "If the definition occurs in a function block, the scope extends to any blocks contained within the defining one, unless a contained block introduces a different binding for the name." There's a distinction between local and available. – Matthew Trevor Nov 22 '16 at 08:35
  • @PM2Ring "If the definition occurs in a function block, the scope extends to any blocks contained within the defining one" `outer` is a function block, `inner` is a function block contained within it, why isn't the scope of `outer` extending to `inner` as described? – Matthew Trevor Nov 22 '16 at 08:45
  • Why don't you use your **actual** code, the `locals` behaviour is *different* from your actual issue. Just try printing `locals()['x']` in the `working` function. – Antti Haapala -- Слава Україні Nov 22 '16 at 08:55
  • `x` is not in `locals` within a function as it is not a local variable in that scope. As for the `pdb.set_trace()` within a function - I cannot repeat the name error and thus am unable to reproduce this issue. – Antti Haapala -- Слава Україні Nov 22 '16 at 08:57
  • How exactly were you running this? I have tried copying it into a standard interpreter, running as a script, running in iPython console, importing the functions and running from an interpreter, and in all those cases `x` is visible to the interpreter and is `1`. I'm running on macOS 10.12, and have tried python 3.5.0 and python 2.7.10. Are you sure you saw what you thought you saw? – daphtdazz Nov 22 '16 at 16:55
  • @daphtdazz Sorry, I simplified it to the point where it was no longer a problem. I've updated the question. – Matthew Trevor Nov 22 '16 at 23:42

2 Answers2

2

x is not local to broken ("local" meaning assigned inside the function), so it won't show up in locals().

However, you can access objects from the outer scope if they are not shadowed by a local, so you can access it in working.

Michael Kohl
  • 66,324
  • 14
  • 138
  • 158
1

I think the specification of how this works is covered by the introducing PEP227 (for python 2.1!). Key parts:

The Python 2.0 definition specifies exactly three namespaces ... the local namespace, the global namespace, and the builtin namespace. According to this definition, if a function A is defined within a function B, the names bound in B are not visible in A. The proposal changes the rules so that names bound in B are visible in A (unless A contains a name binding that hides the binding in B).

...

If a name is used within* a code block, but it is not bound there and is not declared global, the use is treated as a reference to the nearest enclosing function region.

...

An analogous function [to locals() and globals()] will not be provided for nested scopes. Under this proposal, it will not be possible to gain dictionary-style access to all visible scopes.

* emphasised bit being the key bit: Your variable has not been used within a code block, so therefore there is nothing to treat as a reference to the enclosing function.

Furthermore, pdb is running dynamically and the default (source code link) action when no command has been specified uses exec with locals and globals from the frame. Therefore the only variables available to it are locals() and globals() of the frame being inspected, which as indicated do not include captured variables from enclosing frames.

daphtdazz
  • 7,754
  • 34
  • 54