Python local variable compile principle

Question

def fun():  
    if False:
        x=3
    print(locals())
    print(x)
fun()

output and error message:

{}
---------------------------------------------------------------------------
UnboundLocalError                         Traceback (most recent call last)
<ipython-input-57-d9deb3063ae1> in <module>()
      4     print(locals())
      5     print(x)
----> 6 fun()

<ipython-input-57-d9deb3063ae1> in fun()
      3         x=3
      4     print(locals())
----> 5     print(x)
      6 fun()

UnboundLocalError: local variable 'x' referenced before assignment

I am wondering how the python interpreter works. Note that x=3 doesn't run at all, and it shouldn't be treated as a local variable, which means the error would be " name 'x' is not defined". But look into the code and the error message, it isn't the case. Could anybody explain the mechanism principle of the compilation of the python interpreter behind this situation?

Answer 1

A name used in a function can have only one scope for the whole function body. The scope is determined at compile time (not when the function is run).

If there's an assignment to a name anywhere in the function (regardless of whether it will be run when the function is called), the compiler will treat that name as local to the function by default. You can use the global and nonlocal statements to explicitly tell it to use a different scope.

A special case is where a name is assigned to in one function's body, and accessed from another function that is defined within the first function. Such a variable will be put in a special closure cell that will be shared between the functions. The outer function will treat the variable like a local, while the inner function can only assign to it if it has a nonlocal statement for the name. Here's an example of a closure and a nonlocal statement:

def counter():
    val = 0
    def helper():
        nonlocal val
        val += 1
        return val
    return helper

In addition to the issue you're seeing, there's another sort of scope confusion that you may see:

x = 1
def foo():
   print(x)  # you might expect this to print the global x, but it raises an exception
   x = 2     # this assignment makes the compiler treat the name x as local to the function

In the foo function the name x is considered a local everywhere, even though the print call tries to use it before it has been assigned to in the local namespace.

Answer 2

So, Python will always categorize each name in each function as being one of local, non-local or global. These name scopes are exclusive; within each function (names in nested functions have their own naming scope), each name can belong to only one of these categories.

When Python compiles this code:

def fun():
    if False:
        x=3

it will result in an abstract syntax tree as in:

FunctionDef(
    name='fun', 
    args=arguments(...), b
    body=[
        If(test=NameConstant(value=False),
            body=[
                Assign(targets=[Name(id='x', ctx=Store())], value=Num(n=3))
            ], 
            orelse=[])
    ]
)

(some stuff omitted for brevity). Now, when this abstract syntax tree is compiled into code, Python will scan over all name nodes. If there is any Name nodes, with ctx=Store(), that name is considered to be local to the enclosing FunctionDef if any, unless overridden with global (i.e. global x) or nonlocal (nonlocal x) statement within the same function definition.

The ctx=Store() will occur mainly when the name in question is used on the left-hand side of an assignment, or as the iteration variable in a for loop.

Now, when Python compiles this to bytecode, the resulting bytecode is

>>> dis.dis(fun)
  4           0 LOAD_GLOBAL              0 (print)
              3 LOAD_FAST                0 (x)
              6 CALL_FUNCTION            1 (1 positional, 0 keyword pair)
              9 POP_TOP
             10 LOAD_CONST               0 (None)
             13 RETURN_VALUE

The optimizer removed the if statement completely; however since the variable was already marked local to the function, LOAD_FAST is used for x, which will result in x being accessed from local variables, and local variables only. Since x has not been set, the UnboundLocalError is thrown. The name print on the other hand was never assigned to, and thus is considered a global name within this function, so its value is loaded with LOAD_GLOBAL.

Answer 3

The fact that x = 3 is unreachable is irrelevant. The function assigns to it, so it must be a local name.

Bear in mind that the whole file is compiled before execution begins, but that the function is defined during the execution phase, when the compiled function definition block is executed, creating the function object.

A sophisticated optimiser could eliminate unreachable code, but CPython's optimiser isn't that smart - it only performs very simple keyhole optimisation.

To get a deeper look at Python internals, take a look at the ast and dis modules.