Debugging Python ANTLR4 Grammar

Question

I'm having an issue with my ANTLR4 grammar not parsing a string correctly. I'm more interested in learning how to solve my problem than solving my specific problem. How can I generate any type of debug information? I want to know what the parser is "thinking" as it parses the string.

The grammar can be found here: https://github.com/Metrink/metrink-fe/blob/master/metrink.g4

I'm using the simple test string: -1d metric('blah', 'blah', 'blah')

I get the following error: 1:2 missing TIME_INDICATOR at 'd'

The grammar defines TIME_INDICATOR as [shmd] so I'm not sure how it's missing a TIME_INDICATOR at the character d when that is one of the possible tokens. What am I missing here?

I'm using Python3 generated from ANTLR4.

link is broken! The correct is `Metrink.g4` (with capitalized M) — Athafoud, Jun 24 '16 at 09:59

score 4 · Accepted Answer · edited May 23 '17 at 10:34

What I usually do is first dump the tokens to see if the actual tokens the parser expects are created.

You can do that with a small test class like this (easily ported to Python):

public class Main {

    static void test(String input) {

        metrinkLexer lexer = new metrinkLexer(new ANTLRInputStream(input));
        CommonTokenStream tokenStream = new CommonTokenStream(lexer);
        tokenStream.fill();

        System.out.printf("input: `%s`\n", input);

        for (Token token : tokenStream.getTokens()) {
            if (token.getType() != TLexer.EOF) {
                System.out.printf("  %-20s %s\n", metrinkLexer.VOCABULARY.getSymbolicName(token.getType()), token.getText());
            }
        }

        System.out.println();
    }

    public static void main(String[] args) throws Exception {
        test("-1d metric('blah', 'blah', 'blah')");
    }
}

If you run the code above, the following will get printed to your console:

input: `-1d metric('blah', 'blah', 'blah')`
  MINUS                -
  INTEGER_LITERAL      1
  IDENTIFIER           d
  METRIC               metric
  LPAREN               (
  STRING_LITERAL       'blah'
  COMMA                ,
  STRING_LITERAL       'blah'
  COMMA                ,
  STRING_LITERAL       'blah'
  RPAREN               )

As you can see, the d is being tokenized as a IDENTIFIER instead of an TIME_INDICATOR. This is because the IDENTIFIER rule is defined before your TIME_INDICATOR rule. The lexer does not "listen" to what the parser might need, it simply matches the most characters as possible, and if two or more rules match the same amount of characters, the rule defined first "wins".

So, d can either be tokenized as TIME_INDICATOR or an IDENTIFIER. If this is dependent on context, I suggest you tokenize it as a IDENTIFIER (and remove TIME_INDICATOR) and create a parser rule like this:

relative_time_literal:
    MINUS? INTEGER_LITERAL time_indicator;

time_indicator:
    {_input.LT(1)getText().matches("[shmd]")}? IDENTIFIER;

The { ... }? is called a predicate: Semantic predicates in ANTLR4?

Also, FALSE and TRUE will need to be placed before the IDENTIFIER rule.

I updated my code to include printing of tokens: https://github.com/Metrink/metrink-fe/commit/2ea0dcede1d5febb37374fa43b056953d434fe66 — wspeirs, May 05 '16 at 18:19

Debugging Python ANTLR4 Grammar

1 Answers1