Why exec() works differently when invoked inside of function and how to avoid it

Question

I'm trying to declare two functions within exec statement in python. Let's call them f1() and f2().

I've found out that when exec is invoked inside of some function, then f2() has no visibility of f1(). However this doesn't happen when exec and function call are placed in global code.

# Case 1: Working fine

code = """
def f1(): print "bar"
def f2(): f1()
"""

exec(code)
f2() # Prints "bar" as expected

# Case 2: Throws NameError: global name 'f1' is not defined

code = """
def f1(): print "bar"
def f2(): f1()
"""

def foo():
    exec(code)
    f2() # NameError

foo()

Can someone explain me how to avoid that NameError and make exec work inside of a function?

Pardon my being flippant, but the best way to avoid problems with `exec` is to never use `exec`. There is almost always a better approach. — Kevin, Apr 19 '19 at 14:13
I know that Python's bytecode compiler has to make certain decisions about the visibility of names at parse time. e.g. whether to emit the instruction `LOAD_GLOBAL`, or `LOAD_DEREF`, or `LOAD_FAST`, etc. `exec` effectively creates its own independent parsing context, so the bytecode compiler can't make a fully informed decision about name resolution, unlike if you had defined each function normally. (Posting this as a comment and not an answer because it's too handwavey to actually give you an idea of what specifically is happening and how to fix it) — Kevin, Apr 19 '19 at 14:45
Related: [*Why does my function that calls exec on marshalled code give an error about globals?*](https://stackoverflow.com/q/12180627/7246614) — timotree, Apr 28 '19 at 17:53
You can refer to [my question](https://stackoverflow.com/q/72335222/9758790) and [my answer](https://stackoverflow.com/a/72340953/9758790). In case one, the `f1()` is added to globals, but in case2, `f1()` is local. So `f1` should be added to `f2.__closure__` in case2. However, in case2 the definition of f2() can not make a correct closures. My quesition and my answer explain why this happen and how to deal with it. Compared with existing answers, my solution avoid making `f1()` global in case2. — hellohawaii, May 26 '22 at 05:57

Mark · Answer 1 · 2019-04-28T17:20:46.407

5

exec() accepts a second parameter for globals. As explained in the docs:

Note The built-in functions globals() and locals() return the current global and local dictionary, respectively, which may be useful to pass around for use as the second and third argument to exec().

So you can make this work by explicitly passing in globals():

code = """
def f1(): print ("bar")
def f2(): f1()
"""

def foo():
    exec(code, globals())
    f2() # works in python2.7 and python3

foo()

If you want to control the scope precisely, you can pass an object into exec:

code = """
def f1(): print ("bar")
def f2(): f1()
"""

def foo():
    context = {}
    exec(code, context)
    context['f2']() 

foo()

edited Apr 28 '19 at 17:20

answered Apr 19 '19 at 14:42

Mark

90,562
7
108
148

It's important to note that the reason this works is not because it's more explicit, it's because this solution makes the global namespace for the `exec`ed code and the local namespace for the `exec`ed code the same so that whenever a local function is defined (like `f1`) it gets copied as a global. – timotree Apr 28 '19 at 17:35

tarkmeper · Answer 2 · 2019-04-19T14:45:52.910

2

Kevin's comment is a good one and bears some repeating - using exec is dangerous there is almost always a better approach.

However, in response to your question in the first case, both f1() and f2() are in the global namespace so when you call f2() it can find f1(). In the second case, they are created in the local space of the foo() function. When f2() is called it can't find the local f1() definition.

You can fix this using:

code = """
global f1
def f1(): print "bar"
def f2(): f1()
"""
def foo():
    exec(code)
    f2()
foo()

Again - this is almost certainly not the way you want to solve this problem.

** EDIT ** Posted wrong version of the code which I had been checking, this version is what I had meant to include.

edited Apr 19 '19 at 14:45

answered Apr 19 '19 at 14:26

tarkmeper

767
3
12

1

`f1` and `f2` are both defined within the same scope (`foo`) why won't `f2` be able to find `f1`? Ans your code doesn't make sense, since there is no global `f1` defined when the `exec` happens – rdas Apr 19 '19 at 14:32
"When f2() is called it can't find the local f1() definition." But why not? Ordinarily, a function defined in a local scope can access any variable defined in that same scope. I agree there is a lookup problem, but it seems to be caused by something other than the usual name resolution rules. – Kevin Apr 19 '19 at 14:32
**global f1** indeed makes code work. But I still don't get it. If both f1 and f2 were defined locally, how come they are not able to see each other? – Myam Apr 19 '19 at 14:36
f1 doesn't have to be defined when `global f1` is called. The statement just indicates the name f1 should be in the global scope. This gets complicated ( part of why it is a bad idea ). I'm not 100% sure here but normally the value f1 gets captured throught the closure [link](https://www.programiz.com/python-programming/closure). When you use exec it doesn't work. See https://stackoverflow.com/questions/2749655/why-are-closures-broken-within-exec for another example of this. – tarkmeper Apr 19 '19 at 14:40
If you disassemble `f2`, you'll see that `exec` is doing something different than if you define `f2` normally inside `foo`; it generates `0 LOAD_GLOBAL 0 (f1)` instead of `0 LOAD_DEREF 0 (f1)`. – chepner Apr 19 '19 at 16:52
1

It think the problem is that at the time `f2` is compiled, `f1` doesn't actually exist in the current scope yet: the *function* does, it just hasn't been bound to a name in `foo`'s scope, which means `f2` gets compiled with `f1` as a global name. – chepner Apr 19 '19 at 17:00

score -1 · Answer 3 · answered Apr 19 '19 at 14:27

-1

"In all cases, if the optional parts [of exec()] are omitted, the code is executed in the current scope."

https://docs.python.org/3.5/library/functions.html#exec

answered Apr 19 '19 at 14:27

JDunken

465
2
7

OP is using python 2, where `exec` is a statement. The behaviour of `exec` in python 2.7 and python 3.x are very different. In python3, the second code posted by OP would give a `NameError` for `f2` itself – rdas Apr 19 '19 at 14:29
1

I don't think scoping entirely explains the outcome here. If you defined `f1` and `f2` normally within `foo`'s body, then f2 would have no problem calling f1. So the problem can't be "functions defined in a non-global scope can't see one another"; this is empirically not the case. – Kevin Apr 19 '19 at 14:31
I realize that op included the python 2.7 tag, but the use of exec as a function in the code example provided would indicate he is using python 3. – JDunken Apr 19 '19 at 14:41
Maybe, maybe not. `exec(code)` is also valid syntax in Python 2.7, for the same reason that `print("Hello, world!")` is valid syntax: superflous parentheses around an expression are a no-op. – Kevin Apr 19 '19 at 14:44
Looking at OP's 'print "bar"', I am wrong about it being python 3, but looks like I was right about passing in the globals, as @MarkMeyer 's answer dose resolve OP's issue. – JDunken Apr 19 '19 at 14:55

Why exec() works differently when invoked inside of function and how to avoid it

3 Answers3

Linked