PHP : Find repeated words with and without space in text

Question

I can find repeated words in text with this function:

$str = 'bob is a good person. mary is a good person. who is the best? are you a good person? bob is the best?';
    function repeated($str)
    {
        $str=trim($str);  
        $str=ereg_replace('[[:space:]]+', ' ',$str);  
        $words=explode(' ',$str);  
        foreach($words as $w)  
        {  
        $wordstats[($w)]++;  
        }  
        foreach($wordstats as $k=>$v)  
        {  
            if($v>=2)  
            {  
                print "$k"." , ";  
            }  
        }  
    }

thats result me like :

bob , good , person , is , a , the , best?

Q : how i can get result repeated words and Multi-part words between space look like :

bob , good , person , is , a , the , best? , good person , is a , a good , is the , bob is

What is the logic behind having `is a`,`a good` etc. as seperate hits? Could they also be groups of three/four/etc.? — Wrikken, Jun 08 '11 at 15:35
_Side note:_ Don't use `ereg_replace()`, as it's deprecated. Use [`preg_replace()`](http://php.net/preg_replace) instead. — Wiseguy, Jun 08 '11 at 16:00
@wiseguy: or even just `preg_split` if it's the only thing it's going to be used for anyway.... — Wrikken, Jun 08 '11 at 16:11
@Wrikken: I want to generate tags from some text.by this way i can generate random tags from text. — Morteza, Jun 09 '11 at 09:37

score 3 · Accepted Answer · answered Jun 08 '11 at 16:07

<?php
$str = 'bob is a good person. mary is a good person. who is the best? are you a good person? bob is the best?';

//all words:
$found = str_word_count(strtolower($str),1);
//get all words with occurance of more then 1
$counts = array_count_values($found);
$repeated = array_keys(array_filter($counts,function($a){return $a > 1;}));
//begin results with the groups of 1 word.
$results = $repeated;
while($word = array_shift($found)){
    if(!in_array($word,$repeated)) continue;
    $additions = array();
    while($add = array_shift($found)){
        if(!in_array($add,$repeated)) break;
        $additions[] = $add;
        $count = preg_match_all('/'.preg_quote($word).'\W+'.implode('\W+',$additions).'/si',$str,$matches);
        if($count > 1){
            $newmatch = $word.' '.implode(' ',$additions);
            if(!in_array($newmatch,$results)) $results[] = $newmatch;
        } else {
            break;
        }
    }
    if(!empty($additions)) array_splice($found,0,0,$additions);
}
var_dump($results);

Yields:

array(17) {
  [0]=>
  string(3) "bob"
  [1]=>
  string(2) "is"
  [2]=>
  string(1) "a"
  [3]=>
  string(4) "good"
  [4]=>
  string(6) "person"
  [5]=>
  string(3) "the"
  [6]=>
  string(4) "best"
  [7]=>
  string(6) "bob is"
  [8]=>
  string(4) "is a"
  [9]=>
  string(9) "is a good"
  [10]=>
  string(16) "is a good person"
  [11]=>
  string(6) "a good"
  [12]=>
  string(13) "a good person"
  [13]=>
  string(11) "good person"
  [14]=>
  string(6) "is the"
  [15]=>
  string(11) "is the best"
  [16]=>
  string(8) "the best"
}

i get error : _syntax error, unexpected T_FUNCTION in_ : `$repeated = array_keys(array_filter($counts,function($a){return $a > 1;}));` — Morteza, Jun 08 '11 at 19:13
It's a PHP5.3 feature, if you use a lesser version, substitute `function($a){return $a > 1;})` with `create_function('$a', 'return $a > 1;')` — Wrikken, Jun 08 '11 at 23:33
thanks, thats work. but how can only show the results? i dont need to put in array. — Morteza, Jun 09 '11 at 09:30
You need the array to keep proper track of duplicates. After the loop, do what you want with it, you can change it to a string with `implode` if you like. — Wrikken, Jun 09 '11 at 09:44

score 2 · Answer 2 · answered Jun 08 '11 at 15:37

can't you just add the double words to the $wordstats array?

$str = 'bob is a good person. mary is a good person. who is the best? are you a good person? bob is the best?';
function repeated($str)
{
    $str=trim($str);  
    $str=ereg_replace('[[:space:]]+', ' ',$str);  
    $words=explode(' ',$str);  
    $lastWord = '';
    foreach($words as $w)  
    {  
        $wordstats[($w)]++;  
        //skip the first loop because that is the only time it should be blank.
        if($lastWord!=''){
            $wordstats[$lastWord.' '.$w]++;
        }
        $lastWord = $w;
    }  
    foreach($wordstats as $k=>$v)  
    {  
        if($v>=2)  
        {  
            print "$k"." , ";  
        }  
    }  
}

I didn't test this, but it should work because it just uses the same technique you are using.

PHP : Find repeated words with and without space in text

2 Answers2

Linked