Variable scope and name resolution in Python

Question

I think that I fundamentally don't understand how Python does things like variable scope and name resolution. In particular, the fact that the function broken() below doesn't work really surprises me. And, though I've fished around the web for awhile looking for a helpful explanation, but I still don't get it. Can anyone explain or link to a good description of how this stuff works in Python, with enough details that it will seem obvious why broken() doesn't work after reading the relevant materials?

# Why does this code work fine
def okay0():
    def foo():
        L = []
        def bar():
            L.append(5)
        bar()
        return L
    foo()

# and so does this
def okay1():
    def foo():
        def bar():
            L.append(5)
        L = []
        bar()
        return L
    foo()

# but the following code raises an exception?
def broken():
    def foo():
        L = []
        bar()
        return L
    def bar():
        L.append(5)
    foo()

# Example
test_list = [okay0, okay1, broken]
for test_function in test_list:
    try:
        test_function()
    except:
        print("broken")
    else:
        print("okay")

Answer 1

A function defined within another function can access its parent's scope.

In your specific case, L is always defined within foo(). On the first two examples, bar() is defined within foo() as well, so it can access L by the rule above (ie, foo() is bar()'s parent).

However, on broken(), bar() and foo() are siblings. They know nothing of each others' scopes, so bar() cannot see L.

From the documentation:

Although scopes are determined statically, they are used dynamically. At any time during execution, there are at least three nested scopes whose namespaces are directly accessible:

• the innermost scope, which is searched first, contains the local names
• the scopes of any enclosing functions, which are searched starting with the nearest enclosing scope, contains non-local, but also non-global names
• the next-to-last scope contains the current module’s global names
• the outermost scope (searched last) is the namespace containing built-in names

Now, why does okay1 work, if L is defined textually after bar()?

Python does not try to resolve the identifiers until it has to actually run the code (dynamic binding, as explained in @Giusti's answer).

When Python gets to execute the function, it sees an identifier L and looks for it on the local namespace. On the cpython implementation, it is an actual dictionary, so it looks on a dictionary for a key named L.

If it does not find it, it checks on the scopes of any enclosing functions, ie the other dictionaries representing the local namespaces of the enclosing functions.

Note that, even if L is defined after bar(), when bar() is called, L has already been defined. So, when bar() is executed, L already exists on the local namespace of foo(), which is searched when Python does not see L within bar().

Supporting piece of the documentation:

A namespace is a mapping from names to objects. Most namespaces are currently implemented as Python dictionaries, but that’s normally not noticeable in any way (except for performance), and it may change in the future.

(...)

The local namespace for a function is created when the function is called, and deleted when the function returns or raises an exception that is not handled within the function. (Actually, forgetting would be a better way to describe what actually happens.) Of course, recursive invocations each have their own local namespace.

A scope is a textual region of a Python program where a namespace is directly accessible. “Directly accessible” here means that an unqualified reference to a name attempts to find the name in the namespace.

Answer 2

It is simpler than it looks.

The first case is probably the most obvious:

 def okay0():
    def foo():
        L = []
        def bar():
            L.append(5)
        bar()
        return L
    foo()

Here all you have are the regular scope rules. L and bar belong to the same scope, and L is declared first. So bar() can access L.

The second sample is also similar:

def okay1():
    def foo():
        def bar():
            L.append(5)
        L = []
        bar()
        return L
    foo()

Here both L and bar() belong to the same scope. They are local to foo(). It may look different because Python uses dynamic binding. That is, the resolution of the name L in foo() is only resolved when the function is called. By that time, Python already knows that L is a local variable to the same function that contains foo(), so the acess is valid.

However, while Python has dynamic binding, it does not have dynamic scope, so this will fail:

def broken():
    def foo():
        L = []
        bar()
        return L
    def bar():
        L.append(5)
    foo()

Here, there are two variables named L. One is local to foo() and another is local to bar(). Since these functions are not nested and Python does not have dynamic scope, they are two different variables. Because bar() does not use L in an assignment, you get an exception.

Answer 3

The broken() function throws the following error:

NameError: name 'L' is not defined

It's because L is defined within foo() and is local to that function. When you try to reference it in some other function such as bar(), it's not going to be defined.

def broken():
    def foo():
        L = []
        bar()
        return L
    def bar():
        L.append(5)
    foo()

Basically, if you declare a variable within a function, it's going to be local to that function....

Answer 4

The line with L = ... in fixed declares L in fixed's scope. (The return before it makes sure the assignment isn't actually executed, just used for scope determination.) The line with nonlocal L declares that L inside foo refers to an outer scope's L, in this case, fixed's. Otherwise, since an assignment to L exists inside foo, it would refer to an L variable inside foo.

Basically:

• An assignment to a variable makes it scoped to the enclosing function.
• A nonlocal or global declaration overrides the scope, instead using the (innermost? outermost?) scope with the variable declared or the global scope, respectively.

def fixed():
    def foo():
        nonlocal L  # Added
        L = []
        bar()
        return L
    def bar():
        L.append(5)
    foo()
    return  # Added
    L = ...  # Added

Answer 5

The most important concept you want to know is environment evaluation model, which is simple but powerful.

Let me offer you a good material.

If you want to read Python document, you can read 4. Execution model — Python 3.7.4 documentation, it is very terse.

When a name is used in a code block, it is resolved using the nearest enclosing scope. The set of all such scopes visible to a code block is called the block’s environment.