Changing lexer behavior in Antlr3?

Question

NAME: ('a'..'z')+;
QUOTED_NAME: Q NAME Q;
Q: '"';

name : NAME | QUOTED_NAME;

for mytext the result is mytext

for "mytext" the result is "mytext"

Is there any way to consider Q when lexing the source string, but eliminate it when QUOTED_NAME's result is requested? So, what I need is:

for mytext the result is mytext

for "mytext" the result is mytext

Please, no inline-Java solutions

score 1 · Accepted Answer · edited May 23 '17 at 10:32

Is there any way to consider Q when lexing the source string, but eliminate it when QUOTED_NAME's result is requested?

No, not without adding target code (Java, or different).

But I don't really see the need: at a later stage you're going to handle the text from your tokens, why can't you remove the quotes at that stage and leave them in there at the lexing-stage?

You could let the parser create a quoted name and then letting the parser produce an AST instead of a simple parse tree. With an AST you can tell the parser which tokens to include (and which to remove) from the AST. You can then remove the quotes from your tree. However, handling quoted text is realy more a task that belongs to the lexer.

Anyway, here's a small demo of how to create an AST that removes the quotes from the tree:

grammar Test;

options {
  output=AST;
}

parse
 : name+ EOF -> name+
 ;

name
 : NAME 
 | quoted_name
 ;

quoted_name
 : Q NAME Q -> NAME
 ;

NAME  : ('a'..'z')+;
Q     : '"';
SPACE : ' ' {$channel=HIDDEN;};

which creates the following AST:

enter image description here

for the input:

"mytext" mytext

More info on creating AST's with ANTLR: How to output the AST built using ANTLR?

Changing lexer behavior in Antlr3?

1 Answers1