Questions tagged [abstract-syntax-tree]

Abstract syntax trees (ASTs) represent the recursive structure of a formal document (program source code).

Abstract Syntax Trees (abbreviated "AST") represent the structure of a formal document (often a computer source program). Nodes in the tree represent syntactically significant chunks of document (function definitions, declarations, statements, expressions and subexpressions). Children of a node represent the parts of that chunk (for a function definition node, the children are likely to be "name", "signature" and "body").

ASTs are widely used in program analysis and transformation systems, as well as classical tools such as compilers.

They are typically constructed by a parsing activity, which is driven by the BNF rules that describe the formal document structure. If one captures precisely how the parser matches the BNF rules to the formal document, the result is a so-called concrete syntax tree (CST). Usually CSTs contain a lot of unnecessary detail (such as parentheses and keywords) that are not needed by the tool using the tree, since the tree nodes themselves essentially represent grammar rules and this information is thus redundant. So parsers often construct ASTs during the parsing process to produce relatively compact trees compared to what a CST would be. See What is the difference between an Abstract Syntax Tree and a Concrete Syntax Tree? for more discussion on this topic.

There is an "ast" tag for StackOverflow; you should use the "abstract-syntax-tree" tag instead.

2945 questions
280
votes
6 answers

Using python's eval() vs. ast.literal_eval()

I have a situation with some code where eval() came up as a possible solution. Now I have never had to use eval() before but, I have come across plenty of information about the potential danger it can cause. That said, I'm very wary about using…
tijko
  • 7,599
  • 11
  • 44
  • 64
212
votes
14 answers

Parse a .py file, read the AST, modify it, then write back the modified source code

I want to programmatically edit python source code. Basically I want to read a .py file, generate the AST, and then write back the modified python source code (i.e. another .py file). There are ways to parse/compile python source code using standard…
Amandasaurus
  • 58,203
  • 71
  • 188
  • 248
126
votes
8 answers

What's the difference between parse trees and abstract syntax trees (ASTs)?

Are they generated by different phases of a compiling process? Or are they just different names for the same thing?
123
votes
9 answers

What is the difference between an Abstract Syntax Tree and a Concrete Syntax Tree?

I've been reading a bit about how interpreters/compilers work, and one area where I'm getting confused is the difference between an AST and a CST. My understanding is that the parser makes a CST, hands it to the semantic analyzer which turns it…
98
votes
6 answers

What kinds of patterns could I enforce on the code to make it easier to translate to another programming language?

I am setting out to do a side project that has the goal of translating code from one programming language to another. The languages I am starting with are PHP and Python (Python to PHP should be easier to start with), but ideally I would be able to…
89
votes
4 answers

How to create AST with ANTLR4?

I've been searching A LOT about this and I couldn't find anything useful that REALLY helps me build an AST. I already know that ANTLR4 doesn't build AST like ANTLR3 used to do. Everyone say: "Hey, use visitors!", but I couldn't find any example or…
Leandro
  • 1,114
  • 1
  • 12
  • 15
86
votes
3 answers

How to construct an abstract syntax tree

I have a general idea of what an AST is, but I want to know how to construct one. If you're given a grammar and a parse tree, how do you construct the AST? How do you do it if you're given a grammar and an expression?
neuromancer
  • 53,769
  • 78
  • 166
  • 223
76
votes
5 answers

What is JavaScript AST, how to play with it?

Abstract Syntax Tree.. I always heard that compile to SpiderMonkey AST on Github. So, is that a actual standard of JS syntax tree? And there's V8, is V8 using the same kind of AST? How can I play with that?
jiyinyiyong
  • 4,586
  • 7
  • 45
  • 88
62
votes
3 answers

Simple example of how to use ast.NodeVisitor?

Does anyone have a simple example using ast.NodeVisitor to walk the abstract syntax tree in Python 2.6? The difference between visit and generic_visit is unclear to me, and I cannot find any example using google codesearch or plain google.
lacker
  • 5,470
  • 6
  • 36
  • 38
60
votes
0 answers

What's the difference between parse trees and abstract syntax trees?

I found the two terms in a compiler design book, and I'd like to know what each stands for, and how they are different. I searched on the internet and found that parse trees are also called concrete syntax trees (CSTs).
Shashi Bhushan
  • 1,481
  • 3
  • 16
  • 20
58
votes
4 answers

TypeScript: get syntax tree

I had read "whole internet", but can't find any examples about getting syntax tree (just like in Esprima) from TypeScrypt source. I mean how can i get object like this (Esprima Parser example) { "type": "Program", "body": [ { …
bukvaG
  • 597
  • 1
  • 4
  • 8
56
votes
4 answers

How to view Clang AST?

I am trying to get hold on Clang. So, I would like to view the AST generated by Clang after parsing the given program. Is it possible to dump AST in .dot or .viz format? Is there any tool out there?
username_4567
  • 4,737
  • 12
  • 56
  • 92
56
votes
5 answers

Malformed String ValueError ast.literal_eval() with String representation of Tuple

I'm trying to read in a string representation of a Tuple from a file, and add the tuple to a list. Here's the relevant code. raw_data = userfile.read().split('\n') for a in raw_data : print a btc_history.append(ast.literal_eval(a)) Here is…
50
votes
1 answer

Constructing an Abstract Syntax Tree with a list of Tokens

I want to construct an AST from a list of tokens. I'm making a scripting language and I've already done the lexical analysis part, but I have no idea how to create an AST. So the question is, how do I take something like this: WORD, int WORD,…
metro-man
  • 1,763
  • 2
  • 15
  • 28
48
votes
2 answers

Python 3, Are there any known security holes in ast.literal_eval(node_or_string)?

Are there any known ways for ast.literal_eval(node_or_string)'s evaluation to not actually be safe? If yes, are patches available for them? (I already know about PyPy[sandbox], which is presumably more secure, but unless the answers are yes then no,…
user380772
1
2 3
99 100