regex to ignore duplicate matches

Question

I'm using an application to search this website that I don't have control of right this moment and was wondering if there is a way to ignore duplicate matches using only regex.

Right now I wrote this to get matches for the image source in the pages source code

uses this to retrieve srcs

<span> <img id="imgProduct.*? src="/(.*?)" alt="

from this

<span> <img id="imgProduct_1" class="SmPrdImg selected"     
onclick="(some javascript);" src="the_src_I_want1.jpg" alt="woohee"> </span>
<span> <img id="imgProduct_2" class="SmPrdImg selected"     
onclick="(some javascript);" src="the_src_I_want2.jpg" alt="woohee"> </span>
<span> <img id="imgProduct_3" class="SmPrdImg selected"     
onclick="(some javascript);" src="the_src_I_want3.jpg" alt="woohee"> </span>

the only problem is that the exact same code listed above is duplicated way lower in the source. Is there a way to ignore or delete the duplicates using only regex?

Answer 1

Your pattern's not very good; it's way too specific to your exact source code as it currently exists. As @Truth commented, if that changes, you'll break your pattern. I'd recommend something more like this:

<img[^>]*src=['"]([^'"]*)['"]

That will match the contents of any src attribute inside any <img> tag, no matter how much your source code changes.

To prevent duplicates with regex, you'll need lookahead, and this is likely to be very slow. I do not recommend using regex for this. This is just to show that you could, if you had to. The pattern you would need is something like this (I tested this using Notepad++'s regex search, which is based on PCRE and more robust than JavaScript's, but I'm reasonably sure that JavaScript's regex parser can handle this).

<img[^>]*src=['"]([^'"]*)['"](?!(?:.|\s)*<img[^>]*src=['"]\1['"])

You'll then get a match for the last instance of every src.

The Breakdown

For illustration, here's how the pattern works:

<img[^>]*src=['"]([^'"]*)['"]

This makes sure that we are inside a <img> tag when src comes up, and then makes sure we match only what is inside the quotes (which can be either single or double quotes; since neither is a legal character in a filename anyway we don't have to worry about mixing quote types or escaped quotes).

(?!
    (?:
        .
    |
        \s
    )*
    <img[^>]*src=['"]\1['"]
)

The (?! starts a negative lookahead: we are requiring that the following pattern cannot be matched after this point.

Then (?:.|\s)* matches any character or any whitespace. This is because JavaScript's . will not match a newline, while \s will. Mostly, I was lazy and didn't want to write out a pattern for any possible line ending, so I just used \s. The *, of course, means we can have any number of these. That means that the following (still part of the negative lookahead) cannot be found anywhere in the rest of the file. The (?: instead of ( means that this parenthetical isn't going to be remembered for backreferences.

That bit is <img[^>]*src=['"]\1['"]. This is very similar to the initial pattern, but instead of capturing the src with ([^'"]*), we're referencing the previously-captured src with \1.

Thus the pattern is saying "match any src in an img that does not have any img with the same src anywhere in the rest of the file," which means you only get the last instance of each src and no duplicates.

If you want to remove all instances of any img whose src appears more than once, I think you're out of luck, by the way. JavaScript does not support lookbehind, and the overwhelming majority of regex engines that do wouldn't allow such a complicated lookbehind anyway.

Answer 2

I wouldn't work too hard to make them unique, just do that in the PHP following the preg match with array_unique:

$pattern = '~<span> <img id="imgProduct.*? src="/(.*?)" alt="~is';
$match   = preg_match_all($pattern, $html, $matches);

if ($match)
{
     $matches = array_unique($matches[1]);
}

If you are using JavaScript, then you'd need to use another function instead of array_unique, check PHPJS: http://phpjs.org/functions/array_unique:346