Regular expression for a string literal in flex/lex

Question

I'm experimenting to learn flex and would like to match string literals. My code currently looks like:

"\""([^\n\"\\]*(\\[.\n])*)*"\""        {/*matches string-literal*/;}

I've been struggling with variations for an hour or so and can't get it working the way it should. I'm essentially hoping to match a string literal that can't contain a new-line (unless it's escaped) and supports escaped characters.

I am probably just writing a poor regular expression or one incompatible with flex. Please advise!

Thanks so much everyone! All your comments were very helpful. The regex that has finally worked for me is a variant of the one used in the C specification linked by codadict (and explained by Jonathan): \"(\\(.|\n)|[^\\"\n])*\" — Thomas, Jan 11 '10 at 04:12
Since you found Jonathan's answer helpful, consider adding an upvote for his answer. — codaddict, Jan 11 '10 at 04:17
By the way: nowhere in your question do you specify what language's string literals you're interested in. It's a very good idea to put the language you're asking about in one of the question's tags. — Laurence Gonsalves, Jan 11 '10 at 05:14

score 132 · Answer 1 · edited Mar 02 '20 at 05:55

132

A string consists of a quote mark

followed by zero or more of either an escaped anything

\\.

or a non-quote character, non-backslash character

[^"\\]

and finally a terminating quote

Put it all together, and you've got

\"(\\.|[^"\\])*\"

The delimiting quotes are escaped because they are Flex meta-characters.

edited Mar 02 '20 at 05:55

Dan O

4,323
5
29
44

answered Jan 11 '10 at 03:53

Jonathan Feinberg

44,698
7
80
103

4

This doesn't handle escaping, unfortunately. So this would incorrectly lex `"\""` – Paul Biggar Aug 01 '10 at 12:55
5

You must have missed "zero or more of an escaped anything"? – Jonathan Feinberg Aug 02 '10 at 01:48
6

There are several problems with this answer. First, it's not a valid flex pattern. The leading and trailing double-quotes need to be escaped because otherwise flex treats them as meta-characters. So the pattern should be (perhaps) \"(\\.|[^"])*\" . Second, that pattern still doesn't work. For example, it gets this input wrong: "\\\\" . Third, it doesn't meet the original question's requirement of disallowing newlines. – rob mayoff Jul 27 '11 at 01:13
1

As a regex, this is totally correct. Except for the newline thing, which is easily fixed by replacing `.` with `[^\n]` and `[^"]` with `[^"\n]`. It certainly should match `"\\\\"` too, since the repetition will match the quote `"`, then the escaped slash `\\ `, then the next escaped slash `\\ `, then the terminating quote `"`. The pattern certainly works for me outside of the scope of flex. – d11wtq Nov 15 '11 at 10:38
7

It doesn't matter whether it works outside the scope of flex. The question was about flex. If the lexer produced by flex sees `"\\\\"foo"`, it will match the entire input, instead of just matching the `"\\\\"` part, because the character class doesn't exclude backslashes. – rob mayoff Jan 09 '13 at 04:48
1

@robmayoff is correct. This will incorrectly match all of `"\\"a"` (as: `quote`, `not-quote`, `backslash-dot-anything`, `not-quote`, `quote`). The regex should say `[^"\\]`, not `[^"]`. – Cornstalks Apr 19 '14 at 17:36
There's actually one more subtlety here since . won't match a \n. So the final flex pattern needed with escaping is \"(\\.|\\\n|[^"\\])*\" – Jeff Johnson May 21 '15 at 18:03
1

lol upvoted not b/c it's in flex, but I've been looking for a regex that matches a string literal. – BrockLee Jun 19 '17 at 23:01

score 29 · Answer 2 · answered Feb 13 '12 at 12:30

29

For a single line... you can use this:

\"([^\\\"]|\\.)*\"  {/*matches string-literal on a single line*/;}

answered Feb 13 '12 at 12:30

Pete

299
3
2

1

This is the best answer here. – kirbyfan64sos Jan 15 '16 at 17:47
Shouldn't we make sure that the starting quote is not preceded by a backslash (is escaped) ? – Shrutesh Patil Jan 29 '20 at 20:23
If it is preceded by a backslash, it should match no other rule and thus signal an error. – Noob Doob Mar 05 '21 at 20:37

t0mm13b · Answer 3 · 2014-04-19T17:21:37.597

9

How about using a start state...

int enter_dblquotes = 0;

%x DBLQUOTES
%%

\"  { BEGIN(DBLQUOTES); enter_dblquotes++; }

<DBLQUOTES>*\" 
{ 
   if (enter_dblquotes){
       handle_this_dblquotes(yytext); 
       BEGIN(INITIAL); /* revert back to normal */
       enter_dblquotes--; 
   } 
}
         ...more rules follow...

It was similar to that effect (flex uses %s or %x to indicate what state would be expected. When the flex input detects a quote, it switches to another state, then continues lexing until it reaches another quote, in which it reverts back to the normal state.

edited Apr 19 '14 at 17:21

answered Jan 11 '10 at 04:04

t0mm13b

34,087
8
78
110

2

@Samoz: Not really, it's actually used in languages where string literals are used, it eats up what's between a beginning quote and an end quote, even if there's extra quotes inside it hence the usage of switching states in order to chew up the quotes... – t0mm13b Jun 04 '10 at 23:39
6

The flex manual contains a full example (in terms of flex usage) of parsing C-style strings: http://flex.sourceforge.net/manual/Start-Conditions.html . Search for "quoted strings" on that page. – rob mayoff Jul 27 '11 at 01:22

score 3 · Answer 4 · answered Feb 20 '19 at 15:23

Paste my code snippet about handling string in flex, hope inspire your thinking.

Use Start Condition to handle string literal will be more scalable and clear.

%x SINGLE_STRING

%%

\"                          BEGIN(SINGLE_STRING);
<SINGLE_STRING>{
  \n                        yyerror("the string misses \" to termiate before newline");
  <<EOF>>                   yyerror("the string misses \" to terminate before EOF");
  ([^\\\"]|\\.)*            {/* do your work like save in here */}
  \"                        BEGIN(INITIAL);
  .                         ;
}

How do I save the string to yytext using this method ? – AndreGraveler Jul 08 '21 at 00:43 — AndreGraveler, Jul 08 '21 at 00:43

score 2 · Answer 5 · answered Sep 27 '18 at 00:48

2

This is what we use in Zolang for single line string literals with embedded templates ${...}

\"(\$\{.*\}|\\.|[^\"\\])*\"

answered Sep 27 '18 at 00:48

Þorvaldur Rúnarsson

879
7
15

1

It would be able to match this properly: `"Hello ${some + "world"}"` – Þorvaldur Rúnarsson Sep 27 '18 at 00:49

score 0 · Answer 6 · edited Jun 03 '17 at 20:43

0

An answer that arrives late but which can be useful for the next one who will need it:

\"(([^\"]|\\\")*[^\\])?\"

edited Jun 03 '17 at 20:43

Floern

33,559
24
104
119

answered Jun 03 '17 at 20:31

david

11

4

Welcome to SO. This answer would be improved with text explaining *how* it works and *how* it is different. – Shawn Mehan Jun 03 '17 at 21:03

Regular expression for a string literal in flex/lex

6 Answers6

Linked

Related