Why are module-level variables in a Python exec inaccessible?

Question

I'm trying to use Python's exec in a project to execute embedded Python code.

The problem I've encountered is that variables created at the module-level in an exec statement are inaccessible from functions defined in the same module.

Let's say you have the following Python program:

x = 5
def foo():
    print x
foo()

If you put the above four lines in a file and ran it, it would work no problemo.

However, if you try running this same piece of code from within an exec statement, it won't work.

Here's our previous program, inside an exec statement:

import __builtin__

global_env = {'__builtins__': __builtin__}
local_env = dict()

exec """
x = 5
def foo():
    print x
foo()
""" in global_env, local_env

On execution, instead of working, it yields the following error:

Traceback (most recent call last):
  File "lab.py", line 94, in <module>
    """ in global_env, local_env
  File "<string>", line 5, in <module>
  File "<string>", line 4, in foo
NameError: global name 'x' is not defined

I thought module-level variables were stored globally, but it seems that, at least in exec, they're not.

For example, in the previous example, if you replace the call to foo() with:

print global_env
print local_env

You get:

{'__builtins__': <module '__builtin__' (built-in)>}
{'x': 5, 'foo': <function foo at 0x102c12938>}

So anything defined in the module-level (including x) is stored in locals().

But it's impossible to access x from anywhere except the module-level (of the exec statement). In particular, as we saw above, the local scope of x is invisible to functions defined in the same exec statement.

Workarounds

I've found two ways to workaround this issue, and make x accessible again.

The first one is using the global keyword in the function:

exec """
x = 5
def foo():
    global x
    print x
foo()
""" in global_env, local_env

The second one is using the same dictionary for globals() and locals() in exec:

exec """
x = 5
def foo():
    print x
foo()
""" in global_env

However, these are just half-fixes/workarounds that don't address the original issue.

So my questions are: Why are module-level variables in an exec stored locally, and why are inaccessible from anywhere but the module-level?

Some closely related StackOverflow posts:

• globals and locals in python exec()
• Cannot change global variables in a function through an exec() statement?

Answer 1

To understand what is going on, you need to read the docs very closely. The key part says:

If two separate objects are given as globals and locals, the code will be executed as if it were embedded in a class definition.

This means that local assignments get made into the local namespace (equivalent to class-level variables), but functions (i.e. methods) do not become closures if they try to reference a local (class) variable.

Compare your code with:

class Test(object):
    x = 1
    def foo():
        print x
    foo()

You'll get the same error, for the same reason. foo is not a closure, and so it attempts to reference x in the global namespace (unsuccessfully).

Answer 2

The behaviour you see is well documented:

In all cases, if the optional parts are omitted, the code is executed in the current scope. If only the first expression after in is specified, it should be a dictionary, which will be used for both the global and the local variables. If two expressions are given, they are used for the global and local variables, respectively. If provided, locals can be any mapping object. Remember that at module level, globals and locals are the same dictionary. If two separate objects are given as globals and locals, the code will be executed as if it were embedded in a class definition.

In fact:

In [1]: class Test:
   ...:     x = 5
   ...:     def foo():
   ...:         print(x)
   ...:     foo()
   ...:     
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-1-f20229bce3a1> in <module>()
----> 1 class Test:
      2     x = 5
      3     def foo():
      4         print(x)
      5     foo()

<ipython-input-1-f20229bce3a1> in Test()
      3     def foo():
      4         print(x)
----> 5     foo()
      6 

<ipython-input-1-f20229bce3a1> in foo()
      2     x = 5
      3     def foo():
----> 4         print(x)
      5     foo()
      6 

NameError: name 'x' is not defined

The behaviour you see is what was intended. If you want to execute the code as if it was at module level you must use the same object for globals and locals, so your workaround is what you are supposed to do.

Answer 3

Your question contains the assumption that x is a module-level variable, but in fact it is not.

At module level globals() and locals() are the same (this may be specific to CPython) whereas in function scopes they are different:

def foo():
    print globals() is locals()

print globals() is locals()
foo()

An exec statement where you specify the scope doesn't execute the code at module-level, it executes it with the scope you tell it to execute with. So, your last code snippet is not a half-fix or a workaround, it precisely addresses the issue.

The following also works when it appears at module level. It has different meaning from any of your code since it assigns to x and foo in the module, so it does make x a module-level variable:

exec '''
x = 5
def foo():
    print x
foo()
'''

The reason that last code doesn't work when it appears in a function, and why your first attempt doesn't work, is what Blckknght's answer says. exec with separate scopes executes "as if in a class definition", not "as if in a module" or "as if in a function". That means any functions you define inside it don't access their surrounding namespace. They access their own local namespace and the exec statement's global namespace, neither of which is where x was defined.

Answer 4

This feature of Python has an interesting consequence. If you import a module at the module-level in an exec statement, it will not be accessible from functions defined in the same module either. In other words, this slightly modified example from the question will not work, for the same reason(s) explained in the answers:

exec """
import x
def foo():
    print x
foo()
""" in global_env, local_env

Both workarounds provided in the question can be applied to fix this