Extracting cyrillic terms/keywords from text in php

Question

I'm trying to build keywords for my webpage and I want that keywords to be extracted from text. I have that function

function extractCommonWords($string){
     $stopWords = array('и', 'или');

      $string = preg_replace('/ss+/i', '', $string);
      $string = trim($string); 
      $string = preg_replace('/[^a-zA-Z0-9 -]/', '', $string); 
      $string = strtolower($string); 
      preg_match_all('/\b.*?\b/i', $string, $matchWords);
      $matchWords = $matchWords[0];

      foreach ( $matchWords as $key=>$item ) {
          if ( $item == '' || in_array(strtolower($item), $stopWords) || strlen($item) <= 3 ) {
              unset($matchWords[$key]);
          }
      }  
      $wordCountArr = array();
      if ( is_array($matchWords) ) {
          foreach ( $matchWords as $key => $val ) {
              $val = strtolower($val);
              if ( isset($wordCountArr[$val]) ) {
                  $wordCountArr[$val]++;
              } else {
                  $wordCountArr[$val] = 1;
              }
          }
      }
      arsort($wordCountArr);
      $wordCountArr = array_slice($wordCountArr, 0, 10);
      return $wordCountArr;
}

Here is what I try:

$text = "Текст кирилица";
$words = extractCommonWords($text);
echo implode(',', array_keys($words));

The problem is dosen`t work with cyrillic letters. How to fix that ?

Answer 1

Cyrillic letters are multi-byte characters. You'll need to use multi-byte character function of PHP.

For regular expressions, you'll need to add the /u modifier to make them unicode compliant.

See also Are the PHP preg_functions multibyte safe?

Answer 2

Your pattern to replace will also remove all cyrillic characters, because a-z will not match them.

Add this to the character-class to keep the cyrillic characters:

\p{Cyrillic}

...and use the modifiier u like suggested by GolezTrol.

$string = preg_replace('/[^\p{Cyrillic} a-zA-Z0-9 -]/u', '', $string); 

If you only like to extract cyrillic words, you don't need to replace anything, just use this to match the words:

preg_match_all('/\b(\p{Cyrillic}+)\b/u', $string, $matchWords);