Antlr4 c++ visitor API

Question

I am using Antlr4's C++ visitor api to traverse a parse tree. However, I'm struggling to get it functioning correctly. Namely, I'm not sure how to use the visitChildren(ParseTree *tree) call.

I'm given the context for each rule that I have defined. And I can traverse the tree using the contexts: context->accept[RuleContext]([RuleContext]* rule)

However, when I use those I continually visit the same node multiple times.

For instance:

program:
    : nameRule
      dateRule
      ( statements )*
      EOF
    ;

nameRule
    : NAME IDENTIFIER ;

dateRule
    : DATE IDENTIFIER ;

statements:
    : statementX
    | statementY
    | statementZ
    ;

statementX:
    : // do something here

statementY:
    : // do something here

statementZ:
    : // do something here

IDENTIFIER, DATE, and NAME are terminals.

I build the Antlr parsing structure by the following:

void Parser::parse() {
    ifstream file(FLAGS_c, ifstream::binary);
    // Convert the file into ANTLR's format.
    ANTLRInputStream stream = ANTLRInputStream(file);

    // Give the input to the lexer.
    MyLexer lexer = new MyLexer(&stream);
    // Generate the tokens.
    CommonTokenStream tokens(lexer);

    file.close();

    tokens.fill();

    // Create the translation that will parse the input.
    MyParser parser = new MyParser(&tokens);
    parser->setBuildParseTree(true);
    MyParser::ProgramContext *tree = parser->program();

    auto *visitor = new MyVisitor();
    visitor->visitProgram(tree);
}

So when I try to traverse this it looks similar to this, the class MyVisitor extends MyParserVisitor. MyVisitor is the visitor class I use to traverse the generated tree.

Any MyVisitor::visitProgram(ParserVisitor::ProgramContext *context) {
    this->visitNameRule(context->nameRule());
    this->visitDateRule(context->dateRule());

    if (!this->statements.empty()) {
        for (auto &it : this->statements) {
            this->visitStatements(it);
        }
    }
    return Any(context);
}

// Omitting name and date rules.

Any MyVisitor::visitStatements(ParserVisitor::StatementContext *context) {
    this->visitStatementX(context->statementX());
    this->visitStatementY(context->statementY());
    this->visitStatementZ(context->statementZ());
    return Any(context);
}

In this case, statements X, Y, and Z will be visited every time statements is visited. Even if they aren't present in the input program.

Is this the correct way to use this? If it isn't, then I assume the visitChildren(ParseTree *tree) is the correct api to use at each visitor function. But I don't understand how to get access to the ParseTree data structure from the *Context.

Answer 1

This question is not directly related to the C++ visitor, but a general visitor problem in ANTLR4. What you are doing is to shortcut the visitor walk in a way you are not intended to do. Don't explicitly visit the certain sub trees manually, but instead call the super implementation to let it do for you and collect the result in individual visitStatementXXX functions. Look at this implementation of a (very simple) expression evaluator, used in a unit test (written in C++). Here's a partial copy to demonstrate the principle:

class EvalParseVisitor : public MySQLParserBaseVisitor {
public:
  std::vector<EvalValue> results; // One entry for each select item.

  bool asBool(EvalValue in) {
    if (!in.isNullType() && in.number != 0)
      return true;
    return false;
  };

  virtual Any visitSelectItem(MySQLParser::SelectItemContext *context) override {
    Any result = visitChildren(context);
    results.push_back(result.as<EvalValue>());
    return result;
  }

  virtual Any visitExprNot(MySQLParser::ExprNotContext *context) override {
    EvalValue value = visit(context->expr());
    switch (value.type) {
      case EvalValue::Null:
        return EvalValue::fromNotNull();
      case EvalValue::NotNull:
        return EvalValue::fromNull();
      default:
        return EvalValue::fromBool(!asBool(value));
    }
  }

  virtual Any visitExprAnd(MySQLParser::ExprAndContext *context) override {
    EvalValue left = visit(context->expr(0));
    EvalValue right = visit(context->expr(1));

    if (left.isNullType() || right.isNullType())
      return EvalValue::fromNull();
    return EvalValue::fromBool(asBool(left) && asBool(right));

    return visitChildren(context);
  }
...

The essential part is the call to visit() which in turn iterates over the child nodes of the given context tree and triggers only visitor functions for elements that actually exist.