Behavior of exec function in Python 2 and Python 3

Question

Following code gives different output in Python2 and in Python3:

from sys import version

print(version)

def execute(a, st):
    b = 42
    exec("b = {}\nprint('b:', b)".format(st))
    print(b)
a = 1.
execute(a, "1.E6*a")

Python2 prints:

2.7.2 (default, Jun 12 2011, 15:08:59) [MSC v.1500 32 bit (Intel)]
('b:', 1000000.0)
1000000.0

Python3 prints:

3.2.3 (default, Apr 11 2012, 07:15:24) [MSC v.1500 32 bit (Intel)]
b: 1000000.0
42

Why does Python2 bind the variable b inside the execute function to the values in the string of the exec function, while Python3 doesn't do this? How can I achieve the behavior of Python2 in Python3? I already tried to pass dictionaries for globals and locals to exec function in Python3, but nothing worked so far.

--- EDIT ---

After reading Martijns answer I further analyzed this with Python3. In following example I give the locals() dictionay as d to exec, but d['b'] prints something else than just printing b.

from sys import version

print(version)

def execute(a, st):
    b = 42
    d = locals()
    exec("b = {}\nprint('b:', b)".format(st), globals(), d)
    print(b)                     # This prints 42
    print(d['b'])                # This prints 1000000.0
    print(id(d) == id(locals())) # This prints True
a = 1.
execute(a, "1.E6*a")

3.2.3 (default, Apr 11 2012, 07:15:24) [MSC v.1500 32 bit (Intel)]
b: 1000000.0
42
1000000.0
True

The comparison of the ids of d and locals() shows that they are the same object. But under these conditions b should be the same as d['b']. What is wrong in my example?

Answer 1

There is a big difference between exec in Python 2 and exec() in Python 3. You are treating exec as a function, but it really is a statement in Python 2.

Because of this difference, you cannot change local variables in function scope in Python 3 using exec, even though it was possible in Python 2. Not even previously declared variables.

locals() only reflects local variables in one direction. The following never worked in either 2 or 3:

def foo():
    a = 'spam'
    locals()['a'] = 'ham'
    print(a)              # prints 'spam'

In Python 2, using the exec statement meant the compiler knew to switch off the local scope optimizations (switching from LOAD_FAST to LOAD_NAME for example, to look up variables in both the local and global scopes). With exec() being a function, that option is no longer available and function scopes are now always optimized.

Moreover, in Python 2, the exec statement explicitly copies all variables found in locals() back to the function locals using PyFrame_LocalsToFast, but only if no globals and locals parameters were supplied.

The proper work-around is to use a new namespace (a dictionary) for your exec() call:

def execute(a, st):
    namespace = {}
    exec("b = {}\nprint('b:', b)".format(st), namespace)
    print(namespace['b'])

The exec() documentation is very explicit about this limitation:

Note: The default locals act as described for function locals() below: modifications to the default locals dictionary should not be attempted. Pass an explicit locals dictionary if you need to see effects of the code on locals after function exec() returns.

Answer 2

I'd say it's a bug of python3.

def u():
    exec("a=2")
    print(locals()['a'])
u()

prints "2".

def u():
    exec("a=2")
    a=2
    print(a)
u()

prints "2".

But

def u():
    exec("a=2")
    print(locals()['a'])
    a=2
u()

fails with

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 3, in u
KeyError: 'a'

--- EDIT --- Another interesting behaviour:

def u():
    a=1
    l=locals()
    exec("a=2")
    print(l)
u()
def u():
    a=1
    l=locals()
    exec("a=2")
    locals()
    print(l)
u()

outputs

{'l': {...}, 'a': 2}
{'l': {...}, 'a': 1}

And also

def u():
    l=locals()
    exec("a=2")
    print(l)
    print(locals())
u()
def u():
    l=locals()
    exec("a=2")
    print(l)
    print(locals())
    a=1
u()

outputs

{'l': {...}, 'a': 2}
{'l': {...}, 'a': 2}
{'l': {...}, 'a': 2}
{'l': {...}}

Apparently, the action of exec on locals is the following:

• If a variable is set within exec and this variable was a local variable, then exec modifies the internal dictionary (the one returned by locals()) and does not return it to its original state. A call to locals() updates the dictionary (as documented in section 2 of python documentation), and the value set within exec is forgotten. The need of calling locals() to update the dictionary is not a bug of python3, because it is documented, but it is not intuitive. Moreover, the fact that modifications of locals within exec don't change the locals of the function is a documented difference with python2 (the documentation says "Pass an explicit locals dictionary if you need to see effects of the code on locals after function exec() returns"), and I prefer the behaviour of python2.
• If a variable is set within exec and this variable did not exist before, then exec modifies the internal dictionary unless the variable is set afterwards. It seems that there is a bug in the way locals() updates the dictionary ; this bug gives access to the value set within exec by calling locals() after exec.

Answer 3

To sum it up:

• There is no bug in Python 2 nor in Python 3
• The different behavior of exec stems from exec being a statement in Python 2, while it became a function in Python 3.

Please note:

I do not tell anything new here. This is just an assembly of the truth out there found in all the other answers and comments. All I try here is to bring light to some of the more obscure details.

The only difference between Python 2 and Python 3 is, that, indeed, exec is able to change the local scope of the enclosing function in Python 2 (because it is a statement and can access the current local scope) and cannot do this anymore in Python 3 (because it now is a function, so runs in it's own local scope).

The irritation, however, has nothing to do with the exec statement, it only stems from one special behavior detail:

locals() returns something, which I want to call "a scope-wise mutable singleton which, after the call to locals(), always only references all variables in the local scope".

Please note that the behavior of locals() did not change between Python 2 and 3. So, this behavior together with change of how exec works looks like being erratic, but isn't, as it just exposes some detail, which always was there.

What does "a scope-wise mutable singleton which references variables in local scope" mean?

• It is a scope-wise singleton, as regardless how often you call locals() in the same scope, the object returned is always the same.
  • Hence the observation, that id(d) == id(locals()), because d and locals() refer to the same object, the same singleton, as there can only be one (in a different scope you get a different object, but in the same scope you only see this single one).
• It is mutable, as it is a normal object, so you can alter it.
  • locals() forces all entries in the object to reference the variables in the local scope again.
  • If you change something in the object (via d), this alters the object, as it is a normal mutable object.

• These changes of the singleton do not propagate back into the local scope, because all entries in the object are references to the variables in the local scope. So if you alter entries, these changes the singleton object, and not the contents of where "the references pointed to before you change the reference" (hence you do not alter the local variable).

  • In Python, Strings and Numbers are not mutable. This means, if you assign something to an entry, you do not change the object where the entry points to, you introduce a new object and assign a reference to that to the entry. Example:

    a = 1
    d = locals()
    d['a'] = 300
    # d['a']==300
    locals()
    # d['a']==1
    

  Besides optimization this does:

  • Create new object Number(1) - which is some other singleton, BTW.
  • store pointer to this Number(1) into LOCALS['a']
    (where LOCALS shall be the internal local scope)
  • If not already exist, create SINGLETON object
  • update SINGLETON, so it references all entries in LOCALS
  • store pointer of the SINGLETON into LOCALS['d']
  • Create Number(300), which is not a singleton, BTW.
  • store pointer to these Number(300) into d['a']
  • hence the SINGLETON is updated, too.
  • but LOCALS is not updated, so the local variable a or LOCALS['a'] still is Number(1)
  • Now, locals() is called again, the SINGLETON is updated.
  • As d refers to SINGLETON, not LOCALS, d changes, too!

For more on this surprising detail, why 1 is a singleton while 300 is not, see https://stackoverflow.com/a/306353

But please do not forget: Numbers are immutable, so if you try to change a number to another value, you effectively create another object.

Conclusion:

You cannot bring back the exec behavior of Python 2 to Python 3 (except by changing your code), as there is no way to alter the local variables outside of the program flow anymore.

However, you can bring the behavior of Python 3 to Python 2, such that you, today, can write programs, which run the same, regardless if they run with Python 3 or Python 2. This is because in (newer) Python 2 you can use exec with function like arguments as well (in fact, those is a 2- or 3-tuple), with allows to use the same syntax with the same semantics known from Python 3:

exec "code"

(which only works in Python 2) becomes (which works for Python 2 and 3):

exec("code", globals(), locals())

But beware, that "code" can no more alter the local enclosing scope this way. See also https://docs.python.org/2/reference/simple_stmts.html#exec

Some very last words:

The change of exec in Python 3 is good. Because of optimization.

In Python 2 you were not able to optimize across exec, because the state of all local variables which contained immutable contents could change unpredictably. This cannot happen anymore. Now the usual rules of function invocations apply to exec() like to all other functions, too.

Answer 4

I'm afraid I can't explain it exactly, but it basically comes from the fact that b inside the function is local, and exec() appears to assign to the global b. You'll have to declare b to be global inside the function, and inside the exec statement.

Try this:

from sys import version

print(version)

def execute1(a, st):
    b = 42
    exec("b = {}\nprint('b:', b)".format(st))
    print(b)

def execute2(a, st):
    global b
    b = 42
    exec("global b; b = {}\nprint('b:', b)".format(st))
    print(b)

a = 1.
execute1(a, "1.E6*a")
print()
execute2(a, "1.E6*a")
print()
b = 42
exec("b = {}\nprint('b:', b)".format('1.E6*a'))
print(b)

Which gives me

3.3.0 (default, Oct  5 2012, 11:34:49) 
[GCC 4.4.5]
b: 1000000.0
42

b: 1000000.0
1000000.0

b: 1000000.0
1000000.0

You can see that outside the function, the global b is automatically picked up. Inside the function, you're printing the local b.

Note that I would have thought that exec() always uses the global b first, so that in execute2(), you don't need to declare it inside the exec() function. But I find that doesn't work (which is the part I can't explain exactly).