Eval() with custom dict doesn't work in CPython

Question

I'm trying to run some expressions using a custom dict as globals.

class Namespace(dict):
    def __getitem__(self, key):
        if key == "y":
            return 10
        else:
            return super(Namespace, self).__getitem__(key)

def run_with_dict(d):
    print(eval("x + y", d))
    print(eval("[ (p * y) for p in ['foo', 'bar'] ]", d))
    print(eval("{ p: (p * y) for p in ['foo', 'bar'] }", d))

custom = Namespace()
custom["x"] = 2
regular = {"x": 2, "y": 10}

run_with_dict(regular)
run_with_dict(custom)

When running it in CPython 2.7, it fails only on the map comprehension:

12
['foofoofoofoofoofoofoofoofoofoo', 'barbarbarbarbarbarbarbarbarbar']
{'foo': 'foofoofoofoofoofoofoofoofoofoo', 'bar': 'barbarbarbarbarbarbarbarbarbar'}
12
['foofoofoofoofoofoofoofoofoofoo', 'barbarbarbarbarbarbarbarbarbar']
Traceback (most recent call last):
  File "<stdin>", line 22, in <module>
  File "<stdin>", line 15, in run_with_dict
  File "<string>", line 1, in <module>
  File "<string>", line 1, in <dictcomp>
NameError: global name 'y' is not defined

But when it's run with PyPy 2.7, it works fine. It also works fine in any Python 3.

What implementation difference could explain that? Is this a bug in CPython 2.7 or is it undefined behavior? Is there anything I could do to make it work in both implementations?

I realize that this is toy demonstration code, but is there any reason you couldn't just add `y` as an item to the dict in the initialization? — Mark Ransom, Mar 04 '19 at 17:31
Some variables might be loaded dynamically from disk and it's not know upfront which ones are going to be used in a given eval. — user3204469, Mar 04 '19 at 17:37

Aran-Fey · Answer 1 · 2019-03-04T17:56:46.220

2

CPython often takes shortcuts. The dict comprehension in CPython 2.7 expects the dict to be exactly a dict and not a subclass thereof. It doesn't bother calling your overridden __getitem__ method; it goes straight for dict.__getitem__ which of course can't see an entry with the name y.

I'm not really sure if this is undefined behavior or not, but the fact that's been changed in python 3 would imply that it was a bug.

edited Mar 04 '19 at 17:56

answered Mar 04 '19 at 17:29

Aran-Fey

39,665
11
104
149

If anyone's interested, [a similar thing](https://stackoverflow.com/q/50604799/1222951) happens you pass a subclassed dict to the `FunctionType` constructor. – Aran-Fey Mar 04 '19 at 17:30
2

This cannot be right, because the first two statements work as expected (and do in fact call the overridden `__getitem__`). As noted in the question, it's only the dict comprehension that fails. – ekhumoro Mar 04 '19 at 17:40
@ekhumoro I don't follow your logic there. There's evidently some subtle difference between dict comprehensions and the other tested expressions. Why can't that subtle difference be the way dicts are treated? – Aran-Fey Mar 04 '19 at 17:51
@ekhumoro The first statement executes in the module scope (of eval), in python 2 the second also executes in the module scope ([see this question](https://stackoverflow.com/questions/4198906/list-comprehension-rebinds-names-even-after-scope-of-comprehension-is-this-righ)). The dict comprehension creates a new scope in 2 and 3. It would seem the new scope (in python 2) short circuits the variable lookup and directly uses `dict.__getitem__`. This could be checked of course making making the subclass print out when `__getitem__` is invoked. – Dunes Mar 04 '19 at 17:52
@ekhumoro Ah, I guess I was a bit unclear in my answer... I'll rephrase it a little. – Aran-Fey Mar 04 '19 at 17:54

Eval() with custom dict doesn't work in CPython

1 Answers1