how can I remove duplicate words from arrays?

Question

I have a database query that pulls out strings of text

$descriptionsQuery = mysql_query("select prob_text from opencall where logdatex between $OneHourAgo and $TimeNow ORDER by callref DESC") or die(mysql_error());
$descriptions = array();

while ($row = mysql_fetch_assoc($descriptionsQuery)){
$descriptions[] = $row['prob_text'];
}
//put all the strings together with a space between them
$glue = implode (" ",$descriptions);

What I'm wanting help with is... before the "descriptions[]" are "glued" to be one long string, I'm wanting any duplicate words removing. Once they are glued, I'm relying on there being duplicate words from each original description. It's kind of hard to explain, here's an example of what I mean. 2 users input some text, for example User1: "I have an issue with Leeds server. I am in Leeds" User2: "Margaret in Leeds has a problem, please call margaret". From this, I would like User1 to only have 1 "Leeds" in the final glued string, User2 only have 1 margaret, but both users mention "Leeds" so I would like that there twice in the glued string, once from each user. Is this possible? Any help appreciated.

Why not simply change your SQL query to `SELECT DISTICT prob_text FROM opencall`? — Amal Murali, Jan 08 '14 at 13:37
Please drop mysql_ and move on to mysqli_. It's a matter of time before mysql_ will be dropped by PHP. — Mave, Jan 08 '14 at 13:37
Completely confused by what you want `User1 to only have 1 "Leeds" in the final glued string` and `but both users mention "Leeds" so I would like that there twice in the glued string` contradict each other. I think you need more than just array_unique... Could you try and be a bit more clear with what you want.. — superphonic, Jan 08 '14 at 13:41
If you could show us the out put you get from the query, then give an example of what you want the final string to look like. At the moment we are guessing what the output of the query is. — superphonic, Jan 08 '14 at 13:43
@AmalMurali that's still picking up the same word, could it be case sensitive? — Maff, Jan 08 '14 at 13:45
So you want your final string to read: "I have an issue with Leeds server. am in Margaret has a problem, please call"? — Mathew, Jan 08 '14 at 13:46
@Mave the server uses php4 and isn't going to be upgraded (out of my hands) — Maff, Jan 08 '14 at 13:46
The final output is going in to a word cloud, but we don't want words in there because 1 user has put it in multiple times, only if multiple users have put it in. — Maff, Jan 08 '14 at 13:47
@MatW nearly, I'd want it to be "I have an issue with Leeds server. am in Margaret in Leeds has a problem, please call" - if 2 users say the same word then it's there twice (3 users, 3 times etc), however if 1 user says it twice and another once I only want it there twice, once from each user — Maff, Jan 08 '14 at 13:50

score 5 · Accepted Answer · edited May 23 '17 at 11:49

You can do that with $newarray = array_unique($oldarray).

First explode each of your rows to get an array. Use array_unique() to remove duplicates. Then implode each of your rows, then implode all of them.

$descriptionsQuery = mysql_query("select prob_text from opencall where logdatex between $OneHourAgo and $TimeNow ORDER by callref DESC") or die(mysql_error());
$descriptions = array();

while ($row = mysql_fetch_assoc($descriptionsQuery)){
  $tmp = explode(' ', $row['prob_text']);
  $tmp = array_unique($tmp);
  // or case insensitive
  // $tmp = array_intersect_key($array,array_unique(array_map(strtolower,$array)));
  $descriptions[] = implode(' ', $tmp);
}
//put all the strings together with a space between them
$glue = implode (" ",$descriptions);

http://de3.php.net/function.array-unique

If you want to remove duplicates in a case-insensitive way, you have to change the second line in the while. I found tips here : Best solution to remove duplicate values from case-insensitive array

This is case sensitive, so Margaret and margaret will be in the final string — Mathew, Jan 08 '14 at 14:06

score 1 · Answer 2 · answered Jan 08 '14 at 13:37

1

Better is to do it in the query.

You could do something like

SELECT DISTINCT prob_text FROM opencall WHERE logdatex BETWEEN $OneHourAgo AND $TimeNow ORDER BY callref DESC

This will only select the word once in your database, so you won't select any duplicates.

http://dev.mysql.com/doc/refman/5.0/en/distinct-optimization.html

answered Jan 08 '14 at 13:37

putvande

15,068
3
34
50

1

This is not the question asked :/ – jillro Jan 08 '14 at 13:45

score 0 · Answer 3 · answered Jan 08 '14 at 13:38

use array_unique. or use DISTINCT in the query

$descriptionsQuery = mysql_query("select prob_text from opencall where logdatex between $OneHourAgo and $TimeNow ORDER by callref DESC") or die(mysql_error());
$descriptions = array();

while ($row = mysql_fetch_assoc($descriptionsQuery)){
$descriptions[] = $row['prob_text'];
}

//remove duplicates:
$descriptions = array_unique($descriptions);

//put all the strings together with a space between them
$glue = implode (" ",$descriptions);

Mathew · Answer 4 · 2014-01-08T16:19:25.123

Seems like a good time to use array_walk and anonymous functions. This will filter out all duplicate words in a single message, ignoring case:

// $chat is the db result array
foreach($chat as &$msg) {
    $final = [];
    array_walk(str_word_count($msg, 1), function($word) use (&$final) {
        if (!in_array(strtolower($word), array_map('strtolower', $final))) {
            $final[] = $word;
        }
    });
    $msg = implode(' ', $final);
});        
$filtered = implode(' ', $chat);

Note the use of str_word_count() rather than explode(). I've not tested this in a production env, but it will strip basic punctuation (except ' and -); might be useful when you are trying to create a tag cloud.

how can I remove duplicate words from arrays?

4 Answers4