1

I have a file where I want to ignore parts of it. In the Lexer I use gated semantic predicates to avoid creating tokens for the uninteresting part of the file. My rules are similar to the following.

A 
: {!ignore}?=> 'A' 
;
START_IGNORE
: 'foo' {ignore = true; skip();}
;
END_IGNORE
: 'oof' {ignore = false; skip();}
;
IGNORE
: {ignore}?=> . {skip();}
;

However unless I change START and END to also use semantic predicates (as below) it does not work.. 

A 
: {!ignore}?=> 'A' 
;
START_IGNORE
: {true}?=> 'foo' {ignore = true; skip();}
;
END_IGNORE
: {true}?=> 'oof' {ignore = false; skip();}
;    
IGNORE
: {ignore}?=> . {skip();}
;

Why do I have to add the predicates?

EDIT: I am using antlr-3.4

Heinrich Ody
  • 289
  • 2
  • 9

1 Answers1

1

Why do I have to add the predicates?

You don't. At least, not using ANTLR v3.3. I don't know how exactly you're testing, but don't use ANTLRWorks' interpreter or the Eclipse ANTLR IDE plugin. Always do a little test from the command line.

grammar T;

@parser::members {
  public static void main(String[] args) throws Exception {
    TLexer lexer = new TLexer(new ANTLRStringStream("A foo A B C oof A"));
    TParser parser = new TParser(new CommonTokenStream(lexer));
    parser.parse();
  }
}

@lexer::members {
  private boolean ignore = false;
}

parse
 : (t=. 
     {System.out.printf("[\%02d] type=\%s text='\%s'\n", $t.getCharPositionInLine(), tokenNames[$t.type], $t.text);}
   )* EOF
 ;

A 
 : {!ignore}?=> 'A' 
 ;

START_IGNORE
 : 'foo' {ignore = true; skip();}
 ;

END_IGNORE
 : 'oof' {ignore = false; skip();}
 ;

IGNORE
 : {ignore}?=> . {skip();}
 ;    

SPACE
 : ' ' {skip();}
 ;

Run it like this:

java -cp antlr-3.3.jar org.antlr.Tool T.g
javac -cp antlr-3.3.jar *.java
java -cp .:antlr-3.3.jar TParser

which will print the following:

[00] type=A text='A'
[16] type=A text='A'

I.e.: from the input "A foo A B C oof A" the following: "foo A B C oof" is skipped.

Bart Kiers
  • 166,582
  • 36
  • 299
  • 288
  • I do use the Eclipse plugin to compile etc. but not the interpreter the plugin offers. Compiling it from command-line I got the same result as before, so there are some strange things in my code. Usually I would live with the solution I got, but there are some other issues. I will try to find whats causing the behavior.. – Heinrich Ody Jul 09 '12 at 20:40
  • @HeinrichOdy, in case you're using v3.4, try reverting to v3.3: in my experience, v3.3 has less issues. – Bart Kiers Jul 09 '12 at 20:43
  • Will see whether I can change the version easily and if so, will try 3.3, thanks. – Heinrich Ody Jul 09 '12 at 20:49
  • @HeinrichOdy, well, I showed you that the grammar works with ANTLR 3.3, as you can test yourself. Not sure what else you expect... – Bart Kiers Jul 10 '12 at 11:00