PHP PCRE regex negative lookbehind not working as expected

Question

Edit: None of the supplied links attempt to answer the question of negative lookbehinds not behaving as I would expect, given documentation I've read: https://www.regular-expressions.info/lookaround.html, https://regexr.com/, https://www.pcre.org/original/doc/html/pcrepattern.html#lookbehind

I am attempting to create a regex that will pull naked URL(s) out of a body of text ignoring those within an <a> tag. I have gotten the URL working but when I get down to trying to cancel out results with a negative lookbehind, it is not behaving as I expect. To test what would be found, the second pattern below proves that the preceding pattern match does find href=" or href=' but converting it from a positive lookbehind to a negative lookbehind does not cancel the results. I am sure it is something I am doing, and any help/feedback is greatly appreciated.

For testing, I am using (note the global g difference): https://regexr.com /pattern/gi PHP /pattern/i

Match full URL (works): ((?:\bhttps?:\/\/){0,1}((?:[a-z0-9.\-]+[.][a-z]{2,4}))(?:[^\s()<>{}\[\]]*|\([^\s()]*?\([^\s()]+\)[^\s()]*?\)|\([^\s]+?\))(?:\([^\s()]*?\([^\s()]+\)[^\s()]*?\)|\([^\s]+?\)|[^\s`!()\[\]{};:'".,<>?«»“”‘’]))

Match full URL for only URL's in <a> tag (works): ((?<=(\bhref=("|')))(?:\bhttps?:\/\/){0,1}((?:[a-z0-9.\-]+[.][a-z]{2,4}))(?:[^\s()<>{}\[\]]*|\([^\s()]*?\([^\s()]+\)[^\s()]*?\)|\([^\s]+?\))(?:\([^\s()]*?\([^\s()]+\)[^\s()]*?\)|\([^\s]+?\)|[^\s`!()\[\]{};:'".,<>?«»“”‘’]))

Match full URL for only URL's not in <a> tag (does not work): ((?<!(\bhref=("|')))(?:\bhttps?:\/\/){0,1}((?:[a-z0-9.\-]+[.][a-z]{2,4}))(?:[^\s()<>{}\[\]]*|\([^\s()]*?\([^\s()]+\)[^\s()]*?\)|\([^\s]+?\))(?:\([^\s()]*?\([^\s()]+\)[^\s()]*?\)|\([^\s]+?\)|[^\s`!()\[\]{};:'".,<>?«»“”‘’]))

Body of text being searched through:

Ignore: <a href='google.com'>Google</a>
Convert: https://google.com
Ignore: <a href="https://google.com">Google</a>
Convert: google.com?argu=ment
Ignore: <a href="google.com?argu=ment">Google</a>
Convert: google.com/path/to/script.php
Ignore: <a href='google.com/path/to/script.php'>Google</a>
Convert: https://google.com/path/to/script.php
Ignore: <a href='https://google.com/path/to/script.php'>Google</a>
Convert: google.com/path/to/script.php?argu=ment&and=more
Ignore: <a href="google.com/path/to/script.php?argu=ment&and=more">Google</a>
Convert: https://google.com/path/to/script.php?argu=ment&and=more
Ignore: <a href="https://google.com/path/to/script.php?argu=ment&and=more">Google</a>
Convert: google.com/path/to/script.php?argu=ment&and=more#hash
Ignore: <a href="google.com/path/to/script.php?argu=ment&and=more#hash">Google</a>
Convert: https://google.com/path/to/script.php?argu=ment&and=more#hash
Ignore: <a href="https://google.com/path/to/script.php?argu=ment&and=more#hash">Google</a>
Convert: www.google.com
Ignore: <a href="www.google.com">Google</a>
Convert: https://www.google.com
Ignore: <a href="https://www.google.com">Google</a>
Convert: www.google.com?argu=ment
Ignore: <a href="www.google.com?argu=ment">Google</a>
Convert: www.google.com/path/to/script.php
Ignore: <a href="www.google.com/path/to/script.php">Google</a>
Convert: https://www.google.com/path/to/script.php
Ignore: <a href="https://www.google.com/path/to/script.php">Google</a>
Convert: www.google.com/path/to/script.php?argu=ment&and=more
Ignore: <a href="www.google.com/path/to/script.php?argu=ment&and=more">Google</a>
Convert: https://www.google.com/path/to/script.php?argu=ment&and=more
Ignore: <a href="https://www.google.com/path/to/script.php?argu=ment&and=more">Google</a>
Convert: www.google.com/path/to/script.php?argu=ment&and=more#hash
Ignore: <a href="www.google.com/path/to/script.php?argu=ment&and=more#hash">Google</a>
Convert: https://www.google.com/path/to/script.php?argu=ment&and=more#hash
Ignore: <a href="https://www.google.com/path/to/script.php?argu=ment&and=more#hash">Google</a>```

PHP from the test script I am using (WARNING: completely unfiltered inputs; do not share a public URL with this code):

<?php

$input_default = <<<INPUT_DEFAULT
Convert: google.com
Ignore: <a href='google.com'>Google</a>
Convert: https://google.com
Ignore: <a href="https://google.com">Google</a>
Convert: google.com?argu=ment
Ignore: <a href="google.com?argu=ment">Google</a>
Convert: google.com/path/to/script.php
Ignore: <a href='google.com/path/to/script.php'>Google</a>
Convert: https://google.com/path/to/script.php
Ignore: <a href='https://google.com/path/to/script.php'>Google</a>
Convert: google.com/path/to/script.php?argu=ment&and=more
Ignore: <a href="google.com/path/to/script.php?argu=ment&and=more">Google</a>
Convert: https://google.com/path/to/script.php?argu=ment&and=more
Ignore: <a href="https://google.com/path/to/script.php?argu=ment&and=more">Google</a>
Convert: google.com/path/to/script.php?argu=ment&and=more#hash
Ignore: <a href="google.com/path/to/script.php?argu=ment&and=more#hash">Google</a>
Convert: https://google.com/path/to/script.php?argu=ment&and=more#hash
Ignore: <a href="https://google.com/path/to/script.php?argu=ment&and=more#hash">Google</a>
Convert: www.google.com
Ignore: <a href="www.google.com">Google</a>
Convert: https://www.google.com
Ignore: <a href="https://www.google.com">Google</a>
Convert: www.google.com?argu=ment
Ignore: <a href="www.google.com?argu=ment">Google</a>
Convert: www.google.com/path/to/script.php
Ignore: <a href="www.google.com/path/to/script.php">Google</a>
Convert: https://www.google.com/path/to/script.php
Ignore: <a href="https://www.google.com/path/to/script.php">Google</a>
Convert: www.google.com/path/to/script.php?argu=ment&and=more
Ignore: <a href="www.google.com/path/to/script.php?argu=ment&and=more">Google</a>
Convert: https://www.google.com/path/to/script.php?argu=ment&and=more
Ignore: <a href="https://www.google.com/path/to/script.php?argu=ment&and=more">Google</a>
Convert: www.google.com/path/to/script.php?argu=ment&and=more#hash
Ignore: <a href="www.google.com/path/to/script.php?argu=ment&and=more#hash">Google</a>
Convert: https://www.google.com/path/to/script.php?argu=ment&and=more#hash
Ignore: <a href="https://www.google.com/path/to/script.php?argu=ment&and=more#hash">Google</a>
INPUT_DEFAULT;
$regex_default = <<<REGEX_DEFAULT
((?:https?:\/\/){0,1}((?:[a-z0-9.\-]+[.][a-z]{2,4}))(?:[^\s()<>{}\[\]]*|\([^\s()]*?\([^\s()]+\)[^\s()]*?\)|\([^\s]+?\))(?:\([^\s()]*?\([^\s()]+\)[^\s()]*?\)|\([^\s]+?\)|[^\s`!()\[\]{};:'".,<>?«»“”‘’]))
REGEX_DEFAULT;
$replace_with_default = '<a target="__" href="$1">$1</a>';
$input = (!empty($_POST['input']) ? $_POST['input'] : $input_default);
$regex = (!empty($_POST['regex']) ? $_POST['regex'] : $regex_default);
$replace_with = (!empty($_POST['replace_with']) ? $_POST['replace_with'] : $replace_with_default);

?>
<form method="POST" action="<?php echo $_SERVER['SCRIPT_NAME']; ?>">
input:<br />
<textarea name="input" style="width: 100%;" rows="10"><?php echo($input); ?></textarea>
<br />
regex:
<table width="100%"><tr>
<td>/</td><td width="100%"><textarea name="regex" style="width: 100%;" rows="1"><?php echo($regex); ?></textarea></td><td>/i</td>
</tr></table>
replace with:<br />
<textarea name="replace_with" style="width: 100%;" rows="1"><?php echo($replace_with); ?></textarea>
<input type="submit" value="Convert »" />
</form>
<?php
if(!empty($_POST)) {
    $replaced_text = preg_replace('/'.$regex.'/i', $replace_with, $input);
    echo('<hr />converted:<br /><textarea style="width: 100%;" rows="20">'.$replaced_text.'</textarea><hr />'.nl2br($replaced_text).'<hr />');
}

Don't use regular expressions to parse HTML, use `DOMDocument`. — Barmar, Aug 17 '23 at 19:50
If you consider that your question is unique for itself, provide a [mre] written in PHP with the test-cases you present and condense the problem statement to the differentiating issue. Apart from that, have you tried using DOM (or Tidy) firsthand? — hakre, Aug 17 '23 at 19:54
@Barmar I am not trying to parse HTML, I am trying to extract URLs from a body of text that may or may not include partial HTML for which I would like to exclude existing tags. The point being to convert URLs to tags that were not already. The pattern works up to the point of excluding matches prefixed with href=" or href='. Just need to know why the negative lookbehind isn't working. — psyjoniz, Aug 17 '23 at 20:12
Regular expressions are generally very poor at distinguishing "inside" and "outside" of patterns. — Barmar, Aug 17 '23 at 20:16
@hakre I have updated my question with a watered down version of the PHP code I am using to test with. I have tested that it works but do not recommend exposing it anywhere publicly. — psyjoniz, Aug 17 '23 at 20:20
Now that's easy: remove all lines starting with "Ignore:" first, then you get the result. Or strip `]*>`, whatever you prefer, see \K etc. how to skip over. You find the PCRE manpage in HTML format here (for the pattern syntax): https://pcre.org/current/doc/html/pcre2pattern.html -- let us known with which part specifically you run into an issue. What is your matching concept? As Barmar wrote, just _inside/outside_ something remains vaguely undefined in terms of the expression itself. — hakre, Aug 17 '23 at 20:29
@hakre I get your point but the body of text could be anything it is a free-form input where users are typically using tags but sometimes they don't and I need to clean it up for them. 'Ignore:' won't be the norm. Looking into the possibility of using \K, though I'm not sure this will be it. Negative lookbehind, as explained, is exactly what I need, it just doesn't seem to behave, even with a simple example ( regexr.com/7ip9v ). But like I said, it's probably a misunderstanding on my part I am just trying to get the point of understanding so I can move on from this requirement. — psyjoniz, Aug 17 '23 at 20:51
What you were looking for is [`(*SKIP)(*F)`](https://stackoverflow.com/questions/24534782/how-do-skip-or-f-work-on-regex). It works [like this](https://regex101.com/r/yQI4OV/2). — bobble bubble, Aug 18 '23 at 11:01
@bobblebubble THANK YOU! The winning regex is: ```((?:]*>(*SKIP)(*F)|https?:\/\/){0,1}((?:[a-z0-9.\-]+[.][a-z]{2,4}))(?:[^\s()<>{}\[\]]*|\([^\s()]*?\([^\s()]+\)[^\s()]*?\)|\([^\s]+?\))(?:\([^\s()]*?\([^\s()]+\)[^\s()]*?\)|\([^\s]+?\)|[^\s`!()\[\]{};:'".,<>?«»“”‘’]))```. Still gotta do a few tests with more variable bodies of text but this works on the example from above. I very much appreciate your time and example. — psyjoniz, Aug 18 '23 at 12:19
You get no points or recognition for closing a question. If they did this action, then they did it for what they felt was the best interests of the site. And do none of the answers in the listed duplicates help answer your question? If not, can you specifically show how they don't as an edit to your question? Doing this is the best way of getting a question re-opened, if you feel that it should be reopened. — Hovercraft Full Of Eels, Aug 18 '23 at 13:57
Also, why are you trying to use regex to parse HTML? There is always some HTML that will break whatever regex you throw at it, and a much better solution is to use an HTML parsing library. — Hovercraft Full Of Eels, Aug 18 '23 at 13:59
@HovercraftFullOfEels I got an alternative method to work. Please read up from here for winning regex. None of the links answer the question about pcre negative lookbehind not behaving as I would expect, given the documentation I've read (https://www.regular-expressions.info/lookaround.html, https://regexr.com/, https://regexr.com/, https://www.pcre.org/original/doc/html/pcrepattern.html#lookbehind). — psyjoniz, Aug 18 '23 at 15:19
I saw what you did, but I'm telling you from experience, that your solution is brittle and will eventually break. Go ahead, keep using your code but do so at your own risk. — Hovercraft Full Of Eels, Aug 18 '23 at 15:21
@HovercraftFullOfEels How I'm choosing to go about solving an issue in the code I'm working on aside, have you bothered yourself to actually look at the issue I've pointed out about negative lookbehinds? It's three fairly simple regex's that should 'pull all', 'pull subset a' 'pull converse of subset a' but does not. Now, that being said, a basic example works (```((?<!foo)bar)```) so I am fairly certain I am misunderstanding how to apply it to a non-basic pattern. I still think it should work, but with a different solution in hand, I'm moving on. Thanks for the responses. — psyjoniz, Aug 18 '23 at 15:42
@psyjoniz You can't do that by use of a negative lookbehind of fixed width. Negative lookbehind of variable width is not supported by PCRE. You could require a whitespace before [using `(?<!\S)`](https://regex101.com/r/7pDZZL/1) but that does not assure from not being inside a `href=""`. Also you could stuff this monster into an atomic group and use a look*ahead* after it to check if there [no closing `>` by `(?![^<]*>)`](https://regex101.com/r/UzXSuK/1). This will be even slower because there is still a lot backtracking vs [using `(*SKIP)(*F)`](https://regex101.com/r/Vrd8l2/1). — bobble bubble, Aug 18 '23 at 20:08
And regarding the discussion whether to parse html using regex, I'm personally just tired of it. One of the *most famous* regex books [has lots of html examples](https://doc.lagout.org/programmation/Regular%20Expressions/Mastering%20Regular%20Expressions_%20Understand%20Your%20Data%20and%20Be%20More%20Productive%20%283rd%20ed.%29%20%5BFriedl%202006-08-18%5D.pdf#%5B%7B%22num%22%3A521%2C%22gen%22%3A0%7D%2C%7B%22name%22%3A%22FitH%22%7D%2C769%5D). :) Short said you can parse strings with regex. If it's your own html... you know your code and know regex, you can parse it. Else prefer a parser. — bobble bubble, Aug 18 '23 at 20:16
The hysteric is probably still because of [the Zalgo thread](https://stackoverflow.com/a/1732454/5527985) which went viral imho mostly because it's funny ([very funny img](https://pbs.twimg.com/media/FoDofkhWAAIKIrn.jpg)). But there is also [discussion and different opinions](https://meta.stackoverflow.com/questions/261561/please-stop-linking-to-the-zalgo-anti-cthulhu-regex-rant). Well, it's a Stack Overflow thing but had a huge impact and made it almost impossible to answer any question that is about regex and html... generally spoken. — bobble bubble, Aug 18 '23 at 20:41
Don't worry, there was nothing wrong with your question. You described the problem very well (easy to understand), provided attempts and showed plenty of examples. I think it's most likely a duplicate [of this question](https://stackoverflow.com/questions/24651869/regex-matching-links-without-a-tag) which I added to the duplicates list. Bear in mind that a duplicate is not something bad, it's just pointing to a resource which the current problem can be solved with. — bobble bubble, Aug 18 '23 at 20:57

PHP PCRE regex negative lookbehind not working as expected

0 Answers0