Lexer rule to handle escape of quote with quote or backslash in ANTLR4?

Question

I'm trying to expand the answer to How do I escape an escape character with ANTLR 4? to work when the " can be escaped both with " and \. I.e. both

"Rob ""Commander Taco"" Malda is smart."

and

"Rob \"Commander Taco\" Malda is smart."

are both valid and equivalent. I've tried

StringLiteral : '"' ('""'|'\\"'|~["])* '"';

but if fails to match

"Entry Flag for Offset check and for \"don't start Chiller Water Pump Request\""

with the tokenizer consuming more characters than intended, i.e. consumes beyond \""

Anyone who knows how to define the lexer rule?

---

A bit more detail...

• "" succeeds
• """" succeeds
• \" " succeeds
• "\"" succeeds (at EOF)
• "\""\n"" fails (it greedily pulls in the \n and "

Example: (text.txt)

""
""""
"\" "
"\""
""

grun test tokens -tokens < test.txt

line 5:1 token recognition error at: '"'
[@0,0:1='""',<StringLiteral>,1:0]
[@1,2:2='\n',<'
'>,1:2]
[@2,3:6='""""',<StringLiteral>,2:0]
[@3,7:7='\n',<'
'>,2:4]
[@4,8:12='"\" "',<StringLiteral>,3:0]
[@5,13:13='\n',<'
'>,3:5]
[@6,14:19='"\""\n"',<StringLiteral>,4:0]
[@7,21:20='<EOF>',<EOF>,5:2]

\"" and """ at the end of a StringListeral are not being handled the same.

Here's the ATN for that rule:

From this diagram it's not clear why they should be handled differently. They appear to be parallel constructs.

---

More research

Test Grammar (small change to simplify ATN):

grammar test
    ;

start: StringLiteral (WS? StringLiteral)+;

StringLiteral: '"' ( (('\\' | '"') '"') | ~["])* '"';
WS:            [ \t\n\r]+;

The ATN for StringLiteral in this grammar:

OK, let's walk through this ATN with the input "\""\n"

unconsumed input	transition
"\""\n"	1 -ε-> 5
"\""\n"	5 -"-> 11
\""\n"	11 -ε-> 9
\""\n"	9 -ε-> 6
\""\n"	6 -\-> 7
""\n"	7 -"-> 10
"\n"	10 -ε-> 13
"\n"	13 -ε-> 11
"\n"	11 -ε-> 12
"\n"	12 -ε-> 14
"\n"	14 -"-> 15
\n"	15 -ε-> 2

We should reach State 2 with the " before the \n, which would be the desired behavior.

Instead, we see it continue on to consume the \n and the next "

line 2:1 token recognition error at: '"'
[@0,0:5='"\""\n"',<StringLiteral>,1:0]
[@1,7:6='<EOF>',<EOF>,2:2]

In order for this to be valid, there must be a path from state 11 to state 2 that consumes a \n and a " (and I'm not seeing it)

Maybe I'm missing something, but it's looking more and more like a bug to me.

Answer 1

I cannot reproduce it. Given the grammar:

grammar T;

parse
 : .*? EOF
 ;

StringLiteral
 : '"' ( '""' | '\\"' | ~["] )* '"'
 ;

Other
 : . -> skip
 ;

The following code:

String source =
    "\"Rob \"\"Commander Taco\"\" Malda is smart.\"\n" +
    "\"Rob \\\"Commander Taco\\\" Malda is smart.\"\n" +
    "\"Entry Flag for Offset check and for \\\"don't start Chiller Water Pump Request\\\"\"\n";

TLexer lexer = new TLexer(CharStreams.fromString(source));
CommonTokenStream stream = new CommonTokenStream(lexer);

stream.fill();

for (Token t : stream.getTokens()) {
    System.out.printf("%-20s '%s'\n",
        TLexer.VOCABULARY.getSymbolicName(t.getType()),
        t.getText().replace("\n", "\\n"));
}

produces the following output:

StringLiteral        '"Rob ""Commander Taco"" Malda is smart."'
StringLiteral        '"Rob \"Commander Taco\" Malda is smart."'
StringLiteral        '"Entry Flag for Offset check and for \"don't start Chiller Water Pump Request\""'

Tested with ANTLR 4.9.3 and 4.10.1: both produce the same output.

Answer 2

The problem is handling the \ properly.

Bart found the path through the ATN that I missed and allowed it to match the extra \n". The \ is matched as a ~["] and then comes back through and matches the " to terminate the string.

We could disallow \ in the "everything but a " alternative (~["\\]), but then we have to allow a stand-alone \ to be acceptable. We'd want to add an alternative that allows a \ followed by anything other than a ". You'd think that '\\' ~["] does that, and you'd be right, to a point, but it also consumes the character following the \, which is a problem if you want a string like "test \\" string" since it's consumed the second \ you can't match the \" alternative. What you're looking for is a lookahead (i.e. consume the \ if it's not followed by a ", but don't consume the following character). But ANTLR Lexer rules don't allow for lookaheads (ANTLR lexer can't lookahead at all).

You'll notice that most grammars that allow \" as an escape sequence in strings also require a bare \ to be escaped (\\), and frequently treat other \ (other character) sequences as just the "other character").

If escaping the \ character is acceptable, the rule could be simplified to:

StringLiteral: '"' ('\\' . | '""' | ~["\\])* '"';

"Flag for \\"Chiller Water\\"" would not parse correctly, but "Flag for \\\"Chiller Water\\\"" would. Without lookahead, I'm not seeing a way to Lex the first version.

---

Also, note that if you don't escape the \, then you have an ambiguous interpretation of \"". Is it \" followed by a " to terminate the string, or \ followed by "" allowing the string to continue? ANTLR will take whichever interpretation consumes the most input, so we see it using the second interpretation and pulling in characters until if finds a "