0

I've been working with Python's AST (abstract syntax tree) module recently and came upon an issue with it not seeming to run import statements. I couldn't find anything about this and I'm not exactly sure what's going on.

Explanation

When parsing a .py file with AST, and then compiling/running the tree, import statements in the tree are seemingly ignored in favor of those in the script doing the parsing.

Example

Here is a minimal working example of the issue. Made from the extended AST documentation

test.py 

from random import choice

def ast_broken_demo():
    lis = ["foo", "bar"]
    print(choice(lis))

ast_broken_demo()

parser.py 

import ast
#from random import choice

def main():
    path = "test.py"
    source_code = open(path).read()

    a = ast.parse(source_code)
    ast.fix_missing_locations(a)
    co = compile(a, '<ast>', 'exec')
    print(ast.dump(a))
    exec(co)

main()

This results in an error message:

 Traceback (most recent call last): 
 File "test.py", line 18, in <module>
   main()   
 File "test.py", line 16, in main
   exec(co)   
 File "<ast>", line 19, in <module>   
 File "<ast>", line 17, in ast_broken_demo 
 NameError: name 'choice' is not defined

This resolves if you put from random import choice at the top of parser.py, but I shouldn't have to, because that exact line is in test.py and should be executed when the AST runs. Putting the import in parser.py is especially bad because the parser is supposed to parse any arbitrary .py file, so I can't possibly import everything anyone could possibly use.

Whats going on here? Why is the parsing using the wrong imports or not running the import statements inside of test.py? Thank you.

Tryer
  • 120
  • 1
  • 7
  • 1
    Assigning variables (including with `def` or `import`) using `exec` with the default locals is undefined behavior. – user2357112 Oct 15 '18 at 20:17
  • @user2357112 so then the solution is to pass in a scope to exec? Which one? {}, globals(), and locals() all solve the issue. But I'm not sure which is correct, this is new territory for me – Tryer Oct 15 '18 at 20:22
  • 1
    Not locals(), because modifying locals() is undefined behavior. Aside from that, use whatever scope you want to execute the code in. – user2357112 Oct 15 '18 at 20:23
  • @user2357112 Sounds good. If I have this right, I chose simply create an empty namespace `new_namespace = {}` and pass that as the second argument to exec. Exec automatically fills it with the __builtins__, and uses it as the global (or module?) scope for whatever it executes. New local namespaces are created and popped as normal, and at the end, new_namespace contains only the global level variables and function/class references. In my testing so far this doesn't seem to cause any weird scoping issues, so hopefully it stays that way and I actually understand what's going on. – Tryer Oct 15 '18 at 22:20

0 Answers0