Python Regex working different depending on the implementation?

Question

I'm working on a file parser that needs to cut out comments from JavaScript code. The thing is it has to be smart so it won't take '//' sequence inside string as the beggining of the comment. I have following idea to do it:

Iterate through lines. Find '//' sequence first, then find all strings surrounded with quotes ( ' or ") in line and then iterate through all string matches to check if the '//' sequence is inside or outside one of those strings. If it is outside of them it's obvious that it'll be a proper comment begining.

When testing code on following line (part of bigger js file of course):

document.getElementById("URL_LABEL").innerHTML="<a name=\"link\" href=\"http://"+url+"\" target=\"blank\">"+url+"</a>";

I've encountered problem. My regular expression code:

re_strings=re.compile("""   "
                            (?:
                            \\.|
                            [^\\"]
                            )*
                            "
                            |
                            '
                            (?:
                                [^\\']|
                                \\.
                            )*
                            '
                            """,re.VERBOSE);


for s in re.finditer(re_strings,line):
            print(s.group(0))

In python 3.2.3 (and 3.1.4) returns the following strings:

"URL_LABEL"
"<a name=\"
" href=\"
"+url+"
" target=\"
">"
"</a>"

Which is obviously wrong because \" should not exit the string. I've been debugging my regex for quite a long time and it SHOULDN'T exit here. So i used RegexBuddy (with Python compatibility) and Python regex tester at http://re-try.appspot.com/ for reference. The most peculiar thing is they both return same, correct results other than my code, that is:

"URL_LABEL"
"<a name=\"link\" href=\"http://"
"\" target=\"blank\">"
"</a>"

My question is what is the cause of those differences? What have I overlooked? I'm rather a beginer in both Python and regular expressions so maybe the answer is simple...

P.S. I know that finding if the '//' sequence is inside string quotes can be accomplished with one, bigger regex. I've already tried it and met the same problem.

P.P.S I would like to know what I'm doing wrong, why there are differences in behaviour of my code and regex test applications, not find other ideas how to parse JavaScript code.

Answer 1

You just need to use a raw string to create the regex:

re_strings=re.compile(r"""   "
                             etc.
                             "
                        """,re.VERBOSE);

The way you've got it, \\.|[^\\"] becomes the regex \.|[^\"], which matches a literal dot (.) or anything that's not a quotation mark ("). Add the r prefix to the string literal and it works as you intended.

See the demo here. (I also used a raw string to make sure the backslashes appeared in the target string. I don't know how you arranged that in your tests, but the backslashes obviously are present; the problem is that they're missing from your regex.)

Answer 2

you cannot deal with matching quotes with regex ... in fact you cannot guarantee any matching pairs of anything(and nested pairs especially) ... you need a more sophisticated statemachine for that(LLVM, etc...)

source: lots of CS classes...

and also see : Matching pair tag with regex for a more detailed explanation

I know its not what you wanted to hear but its basically just the way it is ... and yes different implementations of regex can return different results for stuff that regex cant really do