2

how can I find the previous/left token in lexer
for example

lexer grammar TLexer;

ID     : [a-zA-Z] [a-zA-Z0-9]*;
CARET  : '^';
RTN    : {someCond1}? CARET ID; // CARET not include this token
GLB    : {someCond2}? CARET ID; // CARET not include this token

etc

DAiMor
  • 3,185
  • 16
  • 24

2 Answers2

5

thanks, I did it this way

lexer grammar TLexer;

@lexer::members {
    int lastTokenType = 0;
public void emit(Token token) {
    super.emit(token);
    lastTokenType = token.getType();
}
}

CARET  : '^';
RTN    : {someCond1&&(lastTokenType==CARET)}? ID;
GLB    : {someCond2&&(lastTokenType==CARET)}? ID;
ID     : [a-zA-Z] [a-zA-Z0-9]*;
DAiMor
  • 3,185
  • 16
  • 24
0

I had a look at the Lexer source. The Lexer answers to nextToken() calls (from the parser). I haven't found that it keeps track of previous tokens. And there is no direct access to CARET. Given this input :

xyz ^abc

and this grammar :

lexer grammar TLexer;

ID     : [a-zA-Z] [a-zA-Z0-9]* {System.out.println("ID ");} ;
CARET  : '^'                   {System.out.println("CARET ");} ;
WS     : [ \r\n] ;
RTN    : CARET ID {System.out.println("RTN " + _tokenStartCharIndex);} ;

the output is :

$ antlr4 TLexer.g4 
$ javac TLexer.java 
$ grun TLexer tokens -tokens -diagnostics -trace input.txt 
ID 
RTN 4
[@0,0:2='xyz',<1>,1:0]
[@1,3:3=' ',<3>,1:3]
[@2,4:7='^abc',<4>,1:4]
[@3,8:8='\n',<3>,1:8]
[@4,9:8='<EOF>',<-1>,2:9]

The lexer gives you a single token of type <4> (RTN) for the input ^abc.

BernardK
  • 3,674
  • 2
  • 15
  • 10
  • I know this, but I need another. i need CARET and RTN as two tokens – DAiMor Jan 11 '13 at 20:24
  • @BernardK Link is dead, but the WaybackMachine has [a copy](http://wayback.archive.org/web/20131023064138/http://www.antlr.org/wiki/pages/viewpage.action?pageId=3604497), which looks a lot like [this wiki page](https://theantlrguy.atlassian.net/wiki/pages/viewpage.action?pageId=2687092). – MvG Mar 01 '16 at 12:56