0

I am trying to develop a rather fast full text search. It will read the index, and should ideally run the matching in just one regex.

Therefore, I need a regex that matches lines only if certain words are contained.

E.g. for

my $txt="one two three four five\n".
        "two three four\n".
        "this is just a one two three test\n";

Only line one and three should be matched, since line two does not contain the word "one".

Now I could go through each line in a while() or use multiple regexes, but I need my solution to be fast.

The example from here: http://www.regular-expressions.info/completelines.html ("Finding Lines Containing or Not Containing Certain Words")

is what I need. However, I can't get it to work in Perl. I tried a lot, but it just doesn't come up with any result.

my $txt="one two three four five\ntwo three four\nthis is just a one two three test\n";
my @matches=($txt=~/^(?=.*?\bone\b)(?=.*?\btwo\b)(?=.*?\bthree\b).*$/gi);
print join("\n",@matches);

Gives no output.

In summary: I need a regex to match lines containing multiple words, and returning these whole lines.

Thanks in advance for your help! I tried so much, but just don't get it to work.

Bart Kiers
  • 166,582
  • 36
  • 299
  • 288
Scoox
  • 181
  • 2
  • 11

2 Answers2

1

The ^ and $ meta char by default match only the start- and end of the input. To let them match the start and end of lines, enable the m (MULTI-LINE) flag:

my $txt="one two three four five\ntwo three four\nthis is just a one two three test\n";
my @matches=($txt=~/^(?=.*?\bone\b)(?=.*?\btwo\b)(?=.*?\bthree\b).*$/gim);
print join("\n",@matches);

produces:

one two three four five
this is just a one two three test

But, if you really want a fast search, regex (with a lot of look aheads) is not the way to go, if you ask me.

Bart Kiers
  • 166,582
  • 36
  • 299
  • 288
0

Code:

use 5.012;
use Benchmark qw(cmpthese);
use Data::Dump;
use once;

our $str = <<STR;
one thing
another two
three to go
no war
alone in the dark
war never changes
STR

our @words = qw(one war two);

cmpthese(100000, {
    'regexp with o'             => sub {
        my @m;
        my $words = join '|', @words;
        @m = $str =~ /(?!.*?\b(?:$words)\b)^(.*)$/omg;
        ONCE { say 'regexp with o:'; dd @m }
    },
    'regexp'                    => sub {
        my @m;
        @m = $str =~ /(?!.*?\b(?:@{ [ join '|', @words ] })\b)^(.*)$/mg;
        ONCE { say 'regexp:'; dd @m }
    },
    'while'                     => sub {
        my @m;
        @m = grep $_ !~ /\b(?:@{ [ join '|',@words ] })\b/,(split /\n/,$str);
        ONCE { say 'while:'; dd @m }
    },
    'while with o'              => sub {
        my @m;
        my $words = join '|',@words;
        @m = grep $_ !~ /\b(?:$words)\b/o,(split /\n/,$str);
        ONCE { say 'while with o:'; dd @m }
    }
})

Resulting:

regexp:
("three to go", "alone in the dark")
regexp with o:
("three to go", "alone in the dark")
while:
("three to go", "alone in the dark")
while with o:
("three to go", "alone in the dark")
                 Rate        regexp regexp with o         while  while with o
regexp        19736/s            --           -2%          -40%          -60%
regexp with o 20133/s            2%            --          -38%          -59%
while         32733/s           66%           63%            --          -33%
while with o  48948/s          148%          143%           50%            --

Сonclusion

So, variant with while is a more faster than variant with regexp.``

zloyrusskiy
  • 1
  • 2
  • Hi and thanks! In theory that's right, but your solution does not seem to capture the whole line, but only the matched words. When capturing the whole line, RegExp was about 5% fast.Anyway thank you very much since I learned a lot from your code! – Scoox Jan 21 '11 at 13:50
  • My solution captures whole lines, look more attentive to my code. You can see it in results. – zloyrusskiy Jan 21 '11 at 14:30
  • Sorry, you are right. I mixed that up. However, your query does not match the words. Unfortunately I don't really understand the code, else I'd try to change it. The query should only match items that contain *all* words (e.g. a line "whatever one whatever war two whatever"). Could you be nice enough to change that, even though I misanswered in the first place? Else it looks real promising, on the actual file still 75% increase of speed with while. – Scoox Jan 23 '11 at 08:10
  • Array @words can contain phrases too and work right without changing this code.Write me in "private" what chunks of code you don't understand, i'll try to help you. – zloyrusskiy Jan 23 '11 at 21:18