ANTLR Lexer rule only seems to work as part of parser rule, and not part of another lexer rule

Question

If I have the following the grammar to parse a list of Integers separated by whitespace:

grammar TEST;

test
    : expression* EOF
    ;

expression
    : integerLiteral
    ;

integerLiteral
    : INTLITERAL
    ;

PLUS: '+';
MINUS: '-';

DIGIT: '0'..'9';
DIGITS: DIGIT+;
INTLITERAL: (PLUS|MINUS)? DIGITS;

WS: [ \t\r\n] -> skip;

It does not work! If I pass "100" I get:

line 1:0 extraneous input '100' expecting {<EOF>, INTLITERAL}

However if remove the lexer INTLITERAL rule and place it just under the parser rule integerLiteral like this

integerLiteral
    : (PLUS|MINUS)? DIGITS
    ;

Now it seems to work just fine!

I feel that if I am able to understand why this is I'll begin to understand some idiosyncrasies that I am experiencing.

Answer 1

The lexer creates tokens in the following manner:

1. try to match as many characters for a single token as possible
2. if two tokens match the same characters, let the one defined first "win"

Given the information from the 2 rules above, then you will see that your rules:

DIGITS: DIGIT+;
INTLITERAL: (PLUS|MINUS)? DIGITS;

are the problem. For the input 100 a DIGITS token is created: rule 2 applies here: both rules match 100, but since DIGITS is defined before INTLITERAL, a DIGITS token is created.

Solution 1

Move INTLITERAL above DIGITS:

INTLITERAL: (PLUS|MINUS)? DIGITS;
DIGIT: '0'..'9';
DIGITS: DIGIT+;

But now notice that DIGIT and DIGITS will never become tokens on their own because INTLITERAL will always be matched first. In this case, you can make both of these rules fragments, and then it doesn't matter where you place them because fragment rules are only used inside other lexer rules (not in parser rules)

Solution 2

Make DIGIT and DIGITS fragments

fragment DIGIT: '0'..'9';
fragment DIGITS: DIGIT+;
INTLITERAL: (PLUS|MINUS)? DIGITS;

Solution 3

Or better, don't glue the operator on the INTLITERAL but match it in an unary expression:

expression
    : (MINUS | PLUS) expression
    | expression (MINUS | PLUS) expression
    | integerLiteral
    ;

integerLiteral
    : INTLITERAL
    ;

PLUS: '+';
MINUS: '-';

fragment DIGIT: '0'..'9';

INTLITERAL: DIGIT+;