Remove all duplicate data from a multi-dimensional array

Question

I'm trying to remove all keys with repeated numbers, I've tried some solutions posted here like the one I'm using in my script but none of it worked for my purpose.

This is my array:

There are at least 4 card IDs repeated.

Array
(
    [0] => Array
        (
            [id_card] => 11883834
            [type] => 1
            [registed] => 1547610891
        )

    [1] => Array
        (
            [id_card] => 20311077
            [type] => 1
            [registed] => 1547610891
        )

    [2] => Array
        (
            [id_card] => 16187903
            [type] => 3
            [registed] => 1547610891
        )

    [3] => Array
        (
            [id_card] => 16354099
            [type] => 1
            [registed] => 1547610891
        )

    [4] => Array
        (
            [id_card] => 21133393
            [type] => 4
            [registed] => 1547610891
        )

    [5] => Array
        (
            [id_card] => 15452852
            [type] => 2
            [registed] => 1547610891
        )

    [6] => Array
        (
            [id_card] => 19775869
            [type] => 2
            [registed] => 1547610891
        )

    [7] => Array
        (
            [id_card] => 20311077
            [type] => 1
            [registed] => 1547610891
        )

    [8] => Array
        (
            [id_card] => 21133393
            [type] => 4
            [registed] => 1547610891
        )

    [9] => Array
        (
            [id_card] => 11883834
            [type] => 1
            [registed] => 1547610891
        )

)
1

At the moment I have something like this:

<?php
$array_data = array_map('unserialize', array_unique(array_map('serialize', $myArray)));
echo '<pre>';
print_r($array_data);
echo '</pre>';
?>

With only 10 keys is working perfectly, but when it goes through this with, for example, 50, 100 keys, no longer works.

Any help appreciated.

Answer 1

So the issue here is that the array_unique solutions expect the entire value to be equivalent - they don't just compare based on your ID field.

Instead, you probably want to loop through the array and keep track of which IDs you've seen, and only take those elements for which you haven't seen the ID before.

function filter_duplicate_id_cards($data) {
  $seen = array(); // To keep track of the IDs we've seen
  $filtered = array(); // Will hold the result

  foreach($data as $item) {
    if(array_key_exists($item['id_card'], $seen)) {
      // We already encountered this id card before.
      continue;
    }

    // Never-before seen id card, append it to the result and set the key in $seen
    $filtered[] = $item;
    $seen[$item['id_card']] = TRUE;
  }

  return $filtered;
}

Note that this uses the map form of a PHP array, rather than just appending seen IDs to the list form and using something like in_array to check if we've seen the key. That's important for performance reasons, especially if we're going to be working on large datasets - in_array is O(n) with the # of items in the array whereas array_key_exists is O(1).

A more generic version of this function would look like this:

function filter_duplicate_field($data, $field) {
  $seen = array(); // To keep track of the keys we've seen
  $filtered = array(); // Will hold the result

  foreach($data as $item) {
    if(array_key_exists($item[$field], $seen)) {
      // We already encountered this key before.
      continue;
    }

    // Never-before seen key, append it to the result and set the key in $seen
    $filtered[] = $item;
    $seen[$item[$field]] = TRUE;
  }

  return $filtered;
}

You could then call it like $result = filter_duplicate_field($data, 'id_card');.

Answer 2

This doesn't preserve the original input order, but because the data is indexed so I'll assume that that doesn't matter.

With just 3 calls in a one-liner, you assign temporary associative keys from back to front to eliminate latter duplicates (because php does not permit duplicate keys), then optionally remove the temporary keys with array_values(). No iterated function calls. No lookup arrays.

Code: (Demo)

$array = [
    ['id_card' => 11883834, 'type' => 1, 'registed' => 1547610891],
    ['id_card' => 20311077, 'type' => 1, 'registed' => 1547610891],
    ['id_card' => 16187903, 'type' => 3, 'registed' => 1547610891],
    ['id_card' => 16354099, 'type' => 1, 'registed' => 1547610891],
    ['id_card' => 21133393, 'type' => 4, 'registed' => 1547610891],
    ['id_card' => 15452852, 'type' => 2, 'registed' => 1547610891],
    ['id_card' => 19775869, 'type' => 2, 'registed' => 1547610891],
    ['id_card' => 20311077, 'type' => 1, 'registed' => 1547610891],
    ['id_card' => 21133393, 'type' => 4, 'registed' => 1547610891],
    ['id_card' => 11883834, 'type' => 1, 'registed' => 1547610891]
];

var_export(array_values(array_column(array_reverse($array), null, 'id_card')));

If you change you mind about wanting to keep the first occurrence, you can remove array_reverse(). If the first level keys are irrelevant in the output, you can remove array_values(). These changes would allow the solution to be a one function call task.

p.s. "registered"