How to write a grammar for hex ints in BNFC?

Question

Here's a tiny grammar for hexadecimal integers.

Numbers . Numbers ::= [HexInt];
separator HexInt " " ;

token HexDigit ["0123456789abcdefABCDEF"] ;
rules HexInt ::= "0x" [HexDigit] ;
separator HexDigit "" ;

It fails to parse "0xff", however, because the lexer treats "ff" as a single token. How do I fix that?

Answer 1

There's no easy way to fix it. There seems to be a bug in BNFC that is including the built-in rule for Ident even though your grammar doesn't make use of it, and it takes precedence over HexDigit in your example (longest match wins).

However you can write a token rule for hexadecimals:

token HexInt ({"0x"} ["0123456789abcdefABCDEF"] +) ;

Answer 2

In BNFC, the following declaration says that HexDigits are to be separated by whitespace (not by nothing, as it may seem):

separator HexDigit "" ;