why don't map objects close over their environment?

Question

I am surprised that a map object built with eval does not remember its context. Why is this? The example below includes a (commented out) workaround (and there are others), but that is not what I am looking for. I am trying to understand why this Python behavior is desirable (if it is). I realize this is related to a previous question, but I believe my query is different.

def test():
    x, y, z = 1, 2, 3
    names = 'x', 'y', 'z'
    for s in names:
        print(s, repr(s), eval(s))           #expected
    values = map(eval, names)
    #values = list(values)                   #why is this needed?
    print( [str(v) for v in values] )        #surprising NameError

System info: run with CPython 3.8 under Win 10.

Closures are created when you *define* a function. `eval` isn't being defined here; `map` is simply saving a reference to it. `eval` has no idea that it was wrapped inside a `map` instance when it finally gets called, and `map` has no reason to modify the environment of its function argument. — chepner, May 13 '20 at 20:17
@DeepSpace Blaming `eval` seems wrong unless it is truly unique in this way. What seems problematic here is that the very meaning of an expression is ambiguous when the expression is written. No matter how well you have understood the preceding code, you do not know how the expression will evaluate. This seems undesirable. — Alan, May 18 '20 at 12:50

score 3 · Answer 1 · answered May 13 '20 at 20:25

The issue with your code is not with map, but with the list comprehension you're using to consume the map iterator. A list comprehension creates a function behind the scenes, and uses it to produce the list. However, the anonymous function has its own namespace, and its that namespace that eval is trying to use. That fails, since x, y and z are defined only in the test namespace, and not in the inner function's namespace.

There are a few ways around the issue. You could pass namespaces to eval explicitly, as other answers have suggested. Or you could make the function used by the list comprehension close over the variables you care about, with code like this:

def test():
    x, y, z = 1, 2, 3
    names = 'x', 'y', 'z'
    values = map(eval, names)
    print( [(x, y, z, str(v)) for v in values] ) # anonymous function will be a closure

But by far the best approach is probably to avoid using eval in the first place. It's very precarious to use, and if you're passing it data you can't absolutely trust, it's a massive security risk (since eval will run arbitrary code as long as you can structure it as an expression). For simple expressions like variable names, using dictionary lookups is much better:

def test():
    data = {'x': 1, 'y': 2, 'z': 3}    # use a dictionary, rather than separate variables
    names = 'x', 'y', 'z'

    print([str(data[name]) for name in names])  # either do lookups directly

    values = map(data.get, names)               # or via map
    print([str(v) for v in values])

juanpa.arrivillaga · Answer 2 · 2020-05-13T20:27:56.513

Because eval has no access to the surrounding context. It is a function, that accepts the namespaces that the dynamically executed code will consider global and local. By default, that will be globals() and locals(). The problem is, locals() in map in the list comprehension doesn't contain the variables you have defined.

One solution is to partially apply these arguments:

def test():
    x, y, z = 1, 2, 3
    names = 'x', 'y', 'z'
    for s in names:
        print(s, repr(s), eval(s))
    g, l = globals(), locals()
    values = map(lambda x: eval(x, g, l), names)
    print( [str(v) for v in values] )

And even better solution is to not use eval which you almost certainly do not need to.

Why would map objects close over their environment? That doesn't make sense. What would that even really mean? The function that is being mapped is assumed to have the context it requires, which it normally does, unless you are doing something hackey, like using eval.

Note, Python always uses lexical scoping, and it seems like you assumed it would use dynamic scope. It doesn't. A lot can be said about the relative merits of those two, but it should be enough to note that pretty much all modern languages use lexical scoping. Aside from early Lisps, there is Bash, which in my opinion, is a great example of a bunch of things you want to avoid in any sane language.

For more fun, try `map(lambda x=4, y=5, z=6: eval(x), names)` :) There's your closure. — r.ook, May 13 '20 at 20:26
You ask: "Why would map objects close over their environment?" The answer is: so that the meaning of such expressions will be transparent. It is unclear to me that this would be inconsistent with lexical scoping; you'll have to elaborate. — Alan, May 14 '20 at 14:33
@Alan but what do you even *mean* for `map` to close over the environment? Closures happen *at definition time*. `map` is defined elsewhere, at interpreter start-up. The way `map` works is pretty transparent, what is non-obvious is the way `eval` works, which is subtle about the way `globals()` and `locals()` will work without explicitly passing them. An really, the problem is the list comprehension, as explained in more detail in the other answer, that is what creates *yet another scope* which doesn't contain the `locals()` you thought it would. — juanpa.arrivillaga, May 14 '20 at 18:55

why don't map objects close over their environment?

2 Answers2