Antlr4: Mismatch input

Question

My grammar:

qwe.g4

grammar qwe;

query
    : COLUMN OPERATOR value EOF
    ;

COLUMN
    : [a-z_]+
    ;

OPERATOR
    : ('='|'>'|'<')
    ;

SCALAR
    : [a-z_]+
    ;

value
    : SCALAR
    ;

WS : [ \t\r\n]+ -> skip ;

Handling in Python code:

qwe.py

from antlr4 import InputStream, CommonTokenStream

from qweLexer import qweLexer
from qweParser import qweParser

conditions = 'total_sales>qwe'

lexer = qweLexer(InputStream(conditions))
stream = CommonTokenStream(lexer)
parser = qweParser(stream)
tree = parser.query()


for child in tree.getChildren():
    print(child, '==>', type(child))

Running qwe.py outputs error when parsing (lexing?) value:

How to fix that?

I read some and suppose that there is something to do with COLUMN rule that also matches value...

Answer 1

Your COLUMN and SCALAR lexer rules are identical. When the LExer matches two rules, then the rule that recognizes the longest token will win. If the token lengths are the same (as the are here), the the first rule wins.

Your Token Stream will be COLUMN OPERATOR COLUMN

That's thy the query rule won't match.

As a general practice, it's good to use the grun alias (that the setup tutorial will tell you how to set up) to dump the token stream.

grun qwe tokens -tokens < sampleInputFile

Once that gives you the expected output, you'll probably want to use the grun tool to display parse trees of your input to verify that is correct. All of this can be done without hooking up the generated code into your target language, and helps ensure that your grammar is basically correct before you wire things up.