Do while and while at the same time using ANTLR

Question

Earlier I created this question asking how to create if/else statements using ANTLR 4. I got an excellent answer which also showed how to do while loops. I have implemented this in my language and I am now trying to make a do-while loop using nearly the same principles.

My syntax is as follows for while loops:

count is 0
while count is less than 10
  count+
  if count not equals 10
    write " " + count + ": Getting there..."
  else if count equals 10
    write count + ": The end!"
  end if
end while

And this is what I would like for do-while loops:

count is 0
do
  count+
  write "count is " + count
  if count equals 10
    write "The end!"
  end if
while count is less than 10

I have tested it and they both work, however, not at the same time. Below is my grammar (sorry for posting it all but I think it's necessary).

If my WHILE and END_WHILE tokens are above my DO_WHILE and DO_WHILE_CONDITION tokens, then the while loop works. However, if I switch them around my do-while loop works. If I change the DO_WHILE_CONDITION token to be anything else than while then both works.

Is there anyway that I can have them both work with the current syntax? I understand that it may be a problem because I use the same keyword for multiple things but I hope there is a way to do this.

//////////////////////////////////
// PARSER
//////////////////////////////////

program
 : block EOF
 ;

block
 : (statement (NEW_LINE+ | EOF))*
 ;

statement
 : assignment
 | if_statement
 | while_statement
 | until_statement
 | do_while_statement
 | write
 ;

assignment
 : ID ASSIGN expression # expressionAssignment
 | ID PLUS              # incrementAssignment
 | ID MINUS             # decrementAssignment
 ;

if_statement
 : IF condition_block (ELSE_IF condition_block)* (ELSE NEW_LINE statement_block)? END_IF
 ;

condition_block
 : expression NEW_LINE statement_block
 ;

statement_block
 : block
 ;

while_statement
 : WHILE expression NEW_LINE statement_block END_WHILE
 ;

until_statement
 : UNTIL expression NEW_LINE statement_block END_UNTIL
 ;

do_while_statement
 : DO_WHILE NEW_LINE statement_block DO_WHILE_CONDITION expression
 ;

expression
 : atom                                             # atomExpression
 | expression PLUS expression                       # plusExpression
 | expression MINUS expression                      # minusExpression
 | expression MULTIPLY expression                   # multiplicationExpression
 | expression DIVIDE expression                     # divisionExpression
 | expression PLUS                                  # incrementExpression
 | expression MINUS                                 # decrementExpression
 | expression AND expression                        # andExpression
 | expression OR expression                         # orExpression
 | expression EQUALS expression                     # equalityExpression
 | expression NOT_EQUALS expression                 # notEqualityExpression
 | expression LESS_THAN expression                  # lessThanExpression
 | expression NOT_LESS_THAN expression              # notLessThanExpression
 | expression GREATER_THAN expression               # greaterThanExpression
 | expression NOT_GREATER_THAN expression           # notGreaterThanExpression
 | expression GREATER_THAN_OR_EQUAL expression      # greaterThanOrEqualExpression
 | expression LESS_THAN_OR_EQUAL expression         # lessThanOrEqualExpression
 ;

atom
 : INT                              # integerAtom
 | FLOAT                            # floatAtom
 | BOOLEAN                          # boolAtom
 | ID                               # idAtom
 | STRING                           # stringAtom
 | OPEN_PAR expression CLOSE_PAR    # expressionAtom
 ;

write
 : WRITE expression
 ;

////////////////////////////////// 
// LEXER
//////////////////////////////////

PLUS                        : '+';
MINUS                       : '-';
MULTIPLY                    : '*';
DIVIDE                      : '/';

ASSIGN                      : 'is';
OPEN_CURLY                  : '{';
CLOSE_CURLY                 : '}';
OPEN_PAR                    : '(';
CLOSE_PAR                   : ')';
COLON                       : ':';
NEW_LINE                    : '\r'? '\n';

IF                          : 'if';
ELSE_IF                     : 'else if';
ELSE                        : 'else';
END_IF                      : 'end if';

WHILE                       : 'while';
END_WHILE                   : 'end while';

UNTIL                       : 'until';
END_UNTIL                   : 'end until';

DO_WHILE                    : 'do';
DO_WHILE_CONDITION          : 'while';

EQUALS                      : 'equals';
NOT_EQUALS                  : 'not equals';
LESS_THAN                   : 'is less than';
NOT_LESS_THAN               : 'is not less than';
GREATER_THAN                : 'is greater than';
NOT_GREATER_THAN            : 'is not greater than';
GREATER_THAN_OR_EQUAL       : 'is greater than or equals';
LESS_THAN_OR_EQUAL          : 'is less than or equals';
WRITE                       : 'write';

AND                         : 'and';
OR                          : 'or';
NOT                         : 'not';

BOOLEAN
 : 'TRUE' | 'true' | 'YES' | 'yes'
 | 'FALSE' | 'false' | 'NO' | 'no'
 ;

INT
 : (PLUS | MINUS)? NUMBER+
 ;

FLOAT
 : (PLUS | MINUS)? NUMBER+ ('.' | ',') (NUMBER+)?
 | (PLUS | MINUS)? (NUMBER+)? ('.' | ',') NUMBER+
 ;

NUMBER
 : '0'..'9'
 ;

STRING
 : '"' ( '\\"' | ~["] )* '"'
 ;

ID
 : ('a'..'z' | 'A'..'Z' | '0'..'9')+
 ;

WHITESPACE
 : [ \t]+ -> skip
 ;

COMMENT
 : ( ';;' .*? ';;' | ';' ~[\r\n]* ) -> skip
 ;

score 1 · Accepted Answer · edited May 23 '17 at 12:04

When creating tokens, the lexer does not take into account what the parser might be needing at a certain point. Check this Q&A that describes the rules (for both v3 and v4): Antlr v3 error with parser/lexer rules

This means that in your case, the rule DO_WHILE_CONDITION:

WHILE                       : 'while';
...
DO_WHILE_CONDITION          : 'while';

will never be matched.

Besides that, it's usually not a good idea to "glue" keywords to each other with white spaces. Consider when the input is "end if" (2 spaces). Better create 2 tokens: an END and an IF and use these in your parser rules.

Try something like this:

program
 : block
 ; 

block
 : NEW_LINE* (statement (NEW_LINE+ | EOF))*
 ;

statement
 : assignment
 | if_statement
 | while_statement
 | until_statement
 | do_while_statement
 | write
 ;

assignment
 : ID IS expression # expressionAssignment
 | ID PLUS          # incrementAssignment
 | ID MINUS         # decrementAssignment
 ;

if_statement
 : IF condition_block (ELSE IF condition_block)* (ELSE NEW_LINE statement_block)? END IF
 ;

condition_block
 : expression NEW_LINE statement_block
 ;

statement_block
 : block
 ;

while_statement
 : WHILE expression NEW_LINE statement_block END WHILE
 ;

until_statement
 : UNTIL expression NEW_LINE statement_block END UNTIL
 ;

do_while_statement
 : DO NEW_LINE statement_block WHILE expression
 ;

// Added unary expressions instead of combining them in the lexer.
expression
 : atom                                            # atomExpression
 | MINUS expression                                # unaryMinusExpression
 | PLUS expression                                 # unaryPlusExpression
 | expression PLUS expression                      # plusExpression
 | expression MINUS expression                     # minusExpression
 | expression MULTIPLY expression                  # multiplicationExpression
 | expression DIVIDE expression                    # divisionExpression
 | expression PLUS                                 # incrementExpression
 | expression MINUS                                # decrementExpression
 | expression AND expression                       # andExpression
 | expression OR expression                        # orExpression
 | expression EQUALS expression                    # equalityExpression
 | expression NOT EQUALS expression                # notEqualityExpression
 | expression IS LESS THAN expression              # lessThanExpression
 | expression IS NOT LESS THAN expression          # notLessThanExpression
 | expression IS GREATER THAN expression           # greaterThanExpression
 | expression IS NOT GREATER THAN expression       # notGreaterThanExpression
 | expression IS GREATER THAN OR EQUALS expression # greaterThanOrEqualExpression
 | expression IS LESS THAN OR EQUALS expression    # lessThanOrEqualExpression
 ;

atom
 : INT                              # integerAtom
 | FLOAT                            # floatAtom
 | bool                             # boolAtom
 | ID                               # idAtom
 | STRING                           # stringAtom
 | OPEN_PAR expression CLOSE_PAR    # expressionAtom
 ;

write
 : WRITE expression
 ;

// By making this a parser rule, you needn't inspect the lexer rule 
// to see if it's true or false.
bool
 : TRUE
 | FALSE
 ;

////////////////////////////////// 
// LEXER
//////////////////////////////////

PLUS                        : '+';
MINUS                       : '-';
MULTIPLY                    : '*';
DIVIDE                      : '/';

OPEN_CURLY                  : '{';
CLOSE_CURLY                 : '}';
OPEN_PAR                    : '(';
CLOSE_PAR                   : ')';
COLON                       : ':';
NEW_LINE                    : '\r'? '\n';

IF                          : 'if';
ELSE                        : 'else';
END                         : 'end';
WHILE                       : 'while';
UNTIL                       : 'until';
DO                          : 'do';
EQUALS                      : 'equals';
NOT                         : 'not';
IS                          : 'is';
LESS                        : 'less';
THAN                        : 'than';
GREATER                     : 'greater';
WRITE                       : 'write';
AND                         : 'and';
OR                          : 'or';

TRUE  : 'TRUE'  | 'true'  | 'YES' | 'yes';
FALSE : 'FALSE' | 'false' | 'NO'  | 'no';

INT
 : DIGIT+
 ;

// (DIGIT+)? is the same as: DIGIT*
FLOAT
 : DIGIT+ [.,] DIGIT*
 | DIGIT* [.,] DIGIT+
 ;

// If a rule can never become a token on its own (an INT will always 
// be created instead of a DIGIT), mark it as a 'fragment'.
fragment DIGIT
 : [0-9]
 ;

// Added support for escaped backslashes.
STRING
 : '"' ( '\\"' | '\\\\' | ~["\\] )* '"'
 ;

// Can it start with a digit? Maybe this is better: [a-zA-Z] [a-zA-Z0-9]*
ID
 : [a-zA-Z0-9]+
 ;

WHITESPACE
 : [ \t]+ -> skip
 ;

COMMENT
 : ( ';;' .*? ';;' | ';' ~[\r\n]* ) -> skip
 ;

Which parser both while-constructs without an issue. Also note that I made slight adjustments in your grammar (see the inline comments). The unary expression is an important one, otherwise 1-2 would be tokenized as 2 INT tokens, which could not be matched as an expression in the parser!

Thanks! That solved the problem and that is really some great corrections to the grammar. I had not thought about the need to split tokens which uses a space into multiple tokens as well as the unary expressions. I'm really glad you caught that. I'm not completely sure what exactly solved the while/do-while issue. Was it using the same token for both expressions? — simonbs, Mar 29 '13 at 16:30
@simonbs, the `do-while` parser rule used the token `DO_WHILE_CONDITION`, but this token would never be created by the lexer since the rule `WHILE` matched the same characters *and* it was placed before `DO_WHILE_CONDITION`. — Bart Kiers, Mar 29 '13 at 18:43
Okay, I see. Thanks! On a side note I found that the rules had to be `program : block; block : NEW_LINE* (statement (NEW_LINE+ | EOF))*` Having `EOF` twice made it expect it twice if the input didn't end with a new line. — simonbs, Mar 29 '13 at 19:06
@simonbs, I adjusted my answer accordingly (feel free to edit my answer in such cases!). — Bart Kiers, Mar 29 '13 at 19:47

Do while and while at the same time using ANTLR

1 Answers1