Remove duplicate elements from array using JavaScript

Question

uniqueElementArray= Array.filter(function(itm,i,Array){
                return i==Array.indexOf(itm);
                });

Array contains set of 1000 or even more than 1000 elements. We want to get unique elements from this array. We are using function which is mentioned above finally we are getting unique Elements array but it's creating performance issue. How can i increase the performance. It's taking too much time in IE8 and IE9.w

If JQuery can be an option, you can use $.unique(yourArray). Here is the documentation [Jquery-Unique](http://api.jquery.com/jQuery.unique/) — Ahmet DAL, May 09 '13 at 06:28
What elements does the array contain and is it sorted? You can use setTimeout to process it without locking the main thread If it's sorted and contains (for example) only strings then you can optimise it a little more — HMR, May 09 '13 at 06:29
@AhmetDAL jQuery.unique(array) works on arrays of DOM elements only, not strings or numbers. — Ankit Jaiswal, May 09 '13 at 06:33
Warning for $.unique that it doesn't work on objects: console.log($.unique([{},{}]).length); gives me 2. so does console.log($.unique([{a:1},{a:1}]).length); — HMR, May 09 '13 at 07:10

score 2 · Answer 1 · answered May 09 '13 at 06:40

2

I recommend using underscore.js:

var otherArray = _.uniq(array);

You can see how it's implemented here.

answered May 09 '13 at 06:40

For future visitors; doesn't work on array containing objects: console.log(_.unique([{a:1},{a:1}]).length) will give you 2 – HMR May 09 '13 at 07:14
So would the original slow solution in the question. JS doesn't have a sense of deep equality by default. `_uniq()` does provide an optional `iterator` argument that can help with this. – May 09 '13 at 07:19
Yes, I see the OP uses indexOf and [{a:0}].indexOf({a:0}) returns -1 So the OP is using Array of simple type. The warning was not for the OP but for future stumblers upon that want to use it for an array containing objects. – HMR May 09 '13 at 08:32
The problem I had with IE8 is that it doesn't matter if the array has even a couple of hundred elements. Had to do it asynchronous, not sure if _ works with larger arrays, I'll give it a shot. – HMR May 09 '13 at 08:34
I'll admit I haven't actually tried it on IE8-- pretty sad if it can't handle it. Looking at underscore's implementation, offhand I'd only expect a big speed up if there are lots of duplicate items, so it might not help. One thing you might try is sorting the array first and then using `_uniq(array, true)` – May 09 '13 at 08:37
No, for IE8 no workie. Went to http://underscorejs.org/ pressed F12 in dev tools window under script pasted the code creating 2000 items filled with Math.random() and then _.unique(arr) on it. IE8 gives me "want to stop script" message. I guess the OP is just at the border of still doing synchronously with about 1000 elements. – HMR May 09 '13 at 08:41
Yes, if there are lots of duplicates it's a lot quicker. Instead of filling the items with Math.random I filled them with Math.floor(Math.random()*22) and it worked. It starts bugging out at about 50.000 items. – HMR May 09 '13 at 08:46

score 1 · Answer 2 · edited May 23 '17 at 12:22

1

If you can sort the array (change order of elements), this function seems handy:

unique = function(in_array) {
    var last, result=[];
    in_array.sort();
    for (var i in in_array) if (in_array[i] != last) result.push(last=in_array[i]);
    return result;
}

EDIT:

See also this answer, I think it's great!.

edited May 23 '17 at 12:22

Community

1
1

answered May 09 '13 at 06:50

hegemon

6,614
2
32
30

score 0 · Answer 3 · answered May 09 '13 at 06:29

Use it this way :

var array = Array.prototype.indexOf ?
function(arr, val) {
    return arr.indexOf(val) > -1;
} :

function(arr, val) {
    var i = arr.length;
    while (i--) {
        if (arr[i] === val) {
            return true;
        }
    }
    return false;
}

function removeRedundant(arr, equals) {
  var val, origArr = arr.slice(0);
  arr.length = 0;

  for (var i = 0, len = origArr.length; i < len; ++i) {
    val = origArr[i];
    if (!array(arr, val)) {
        arr.push(val);
    }
  }

  return arr;
}

var arr = [1, 2, 2, 1, 3];
removeRedundant(arr);
console.log(arr); // [1, 2, 3]

score 0 · Answer 4 · answered May 09 '13 at 08:27

In IE8 I've experienced absurdly slow performance when comparing a bunch of floats. In that case asynchronous handling is the only option you have to not get the "script takes too long" message.

IE8 does not have Array.filter function so your code would not work in IE8 anyway.

//create an array with 10.000 floats
var arr=[];
for(i=0;i<10000;i++){
  arr.push(Math.random());
}
//adding one double float to make sure it will be removed
arr.push(arr[5]);
console.log("array created:",arr.length);
var now=new Date();
function getUniqueDone(arr){
    console.log("done asynch",(new Date().getTime())-now);
    console.log("length of array:"+arr.length);//usually is 10.000
}
// works only on arrays containing numbers or strings
function getUnique(arr,callback){
  var i,j,len,counter,
  chuncksOf=1000;//In IE8 I get script warnings at 150 already,
  //  FF and Chrome could handle 1000
  //  depending on what your array contains you can test a bit and
  //  set chuncksOf accordingly. current value does nnot work in IE8
  //  you have to set it to 100 and wait 20 seconds for output
  if(arr.sort){
      // on very large arrays this still might have a problem
      console.log("sorting");
      arr.sort();
      console.log("sort done");
      i=0;
  }else{
      i=arr.i;
      arr=arr.arr;
  } 
  len=arr.length;
  counter=0;
  while(i<len&&counter<chuncksOf){
    for(j=i+1;j<arr.length;j++){
      if(arr[i]===arr[j]){
        arr.splice(j,1);
        j--;
      }
    }
    i++;
    counter++;
  }
  if(i>=len){
      callback(arr);
  }else{
      var pass={
        arr:arr,
        i:i
      }
console.log("i is now:",i);
      setTimeout(function(){
          getUnique(pass,callback);
      },0);
  }
}
console.log("getting unique");
getUnique(arr,getUniqueDone);

Remove duplicate elements from array using JavaScript

4 Answers4

Linked