How to dynamically construct a closure's namespace "pythonically"

Question

I would like to dynamically construct the enclosing environment's namespace of a closure in such a way that the code inside the closure can access this dynamic content. Here is the simplest toy example that illustrated my issue:

def f():
    exec("Y=7",locals())    
    def closure():
        v=eval("Y*2")
        return v  
    return closure

When I use this code, here's what happens:

In [21]: Q = f()

In [22]: Q
Out[22]: <function __main__.closure>

In [23]: Q()
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-23-51e72ed661f9> in <module>()
----> 1 Q()

<ipython-input-20-a47050c4450a> in closure()
      3
      4     def closure():
----> 5         return eval("Y*2")
      6
      7     return closure

<string> in <module>()

NameError: name 'Y' is not defined

So, in this case, the use of "exec" to add the bound variable "Y" to the closure's environment did not work. However, I cannot use a bare exec either, because of the nested function.

I considered using globals() instead of locals():

def f():
    exec("Y=7",globals())

    def closure():
        return eval("Y*2")

    return closure

## -- End pasted text --

In [27]: Q=f()

In [28]: Q
Out[28]: <function __main__.closure>

In [29]: Q()
Out[29]: 14

So, that solves my immediate problem, but what if I make the "exec" block have a mutable value? For example, if I change f() assign different values to Y, we have a new problem:

def f(z):
    exec("Y="+str(z),globals())

    def closure():
        return eval("Y*2")

    return closure

## -- End pasted text --

In [33]: Q=f(2)

In [34]: Q
Out[34]: <function __main__.closure>

In [35]: Q()
Out[35]: 4

In [36]: G=f(4)

In [37]: G()
Out[37]: 8

In [38]: Q()
Out[38]: 8 #doh!

So, what I need is a way to have the bound variable "Y" be located in the namespace of the closure/function environment, not in the "local()" (whatever that is) or "global()", due to above issue.

It seems that "exec" is not able to execute in this particular context. Would I need to exec the closure itself? This seemed to work.

def f(z):
    exec("Y="+str(z),locals())

    exec("def closure():\n return Y*2",locals())

    return eval("closure")

## -- End pasted text --

In [50]: G=f(3)

In [51]: G()
Out[51]: 6

In [52]: Q=f(5)

In [53]: Q()
Out[53]: 10

In [54]: G()
Out[54]: 6

In [55]: Q()
Out[55]: 10 #:-)

Now, what if I have a more complicated object for Y, like a function?

def f(z,args):
  exec("Y= lambda " + args[0] + ","+args[1]+":"+args[0]+"*2+"+args[1]+"*3",locals())

  exec("def closure(x,y):\n return z*Y(x,y)",locals())

  return eval("closure")

## -- End pasted text --

In [64]: Q=f(2,['a','b'])

In [65]: Q(2,2)
Out[65]: 20

In [68]: G=f(3,['a','b'])

In [69]: G(2,-1)
Out[69]: 3

In [70]: Q(2,2)
Out[70]: 20

This seems to work as well.

However, I was wondering if there were a more elegant way to dynamically construct a closure's environment.

I am using IPython with 2.7.11

Python 2.7.11 |Anaconda 2.4.0 (64-bit)| (default, Jan 19 2016, 12:08:31) [MSC v.1500 64 bit (AMD64)]
Type "copyright", "credits" or "license" for more information.

IPython 4.0.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: exec("x=2")

In [2]: x
Out[2]: 2

In [3]:

Answer 1

You cannot do closures this way in Python 2 or 3; no matter how much you'd try to exec, eval and such, the fact that Python compiler does not see a variable assignment in any of the surrounding function scopes of closure means that closure will be looking up Y in global scope instead.

You can verify this using dis.dis, the byte code would use LOAD_GLOBALs to access all of your "closure" variables.

You need to exec the whole code for def f(z):... in one go.

---

The latter 2 examples seem to work, but this one:

def f(z):
    exec("Y="+str(z),locals())
    exec("def closure():\n return Y*2",locals())
    return eval("closure")

would just use the locals() as globals to exec - that is again no proper closure. Instead of closures, you're compiling new function instances that are bound to their own global environment.

The same code could be written in a more fool-proof manner as

def f(z):
    globs = {'Y': z}
    exec('def closure():\n return Y*2', globs)
    return globs['closure']