How to remove duplicates items from the list in javascript

Question

The list is,

{id: 11, type: "sell", quantity: 11, price: 155}
{id: 11, type: "sell", quantity: 11, price: 155}
{id: 11, type: "sell", quantity: 11, price: 155}
{id: 12, type: "buy", quantity: 3, price: 189}
{id: 13, type: "buy", quantity: 4, price: 189}
{id: 14, type: "buy", quantity: 2, price: 189}
{id: 14, type: "buy", quantity: 2, price: 189}

(almost 1000 items) I want to remove duplicates items from the list in javascript, for example, as items of id 11 and 14 having duplicates, so the new list will be after removing,

  {id: 12, type: "buy", quantity: 3, price: 189}
    {id: 13, type: "buy", quantity: 4, price: 189}

The duplicates item will be completely deleted in the new array its not like it will be still there in the new array

So the new array would have all the entries which had no duplicates (if had duplicate, do not display)? — Georgy, Sep 05 '18 at 19:03
Possible duplicate of [Remove duplicates from an array of objects in JavaScript](https://stackoverflow.com/questions/2218999/remove-duplicates-from-an-array-of-objects-in-javascript) — BobRodes, Sep 05 '18 at 19:04
@Georgy yes the new array, would not have any duplicates, completely removed in the new array — Jeff Dean, Sep 05 '18 at 19:15

score 1 · Accepted Answer · answered Sep 05 '18 at 19:29

You could use a Set as closure over the already visited id and remove this object from the result set, if exists.

This approach uses a single loop for the data and for each found duplicate a filtering.

var array = [{ id: 11, type: "sell", quantity: 11, price: 155 }, { id: 11, type: "sell", quantity: 11, price: 155 }, { id: 11, type: "sell", quantity: 11, price: 155 }, { id: 12, type: "buy", quantity: 3, price: 189 }, { id: 13, type: "buy", quantity: 4, price: 189 }, { id: 14, type: "buy", quantity: 2, price: 189 }, { id: 14, type: "buy", quantity: 2, price: 189 }],
    single = array.reduce((s => (r, o) => {
        if (s.has(o.id)) {
            return r.filter(({ id }) => id !== o.id);
        }
        s.add(o.id);
        r.push(o);
        return r;
    })(new Set), []);

console.log(single);

.as-console-wrapper { max-height: 100% !important; top: 0; }

score 0 · Answer 2 · answered Sep 05 '18 at 19:21

A simple solution would be to hash the items after converting them to strings and see if they already exist in the destination array, if not then add them.

String.prototype.hashCode = function() {
  var hash = 0, i, chr;
  if (this.length === 0) return hash;
  for (i = 0; i < this.length; i++) {
    chr   = this.charCodeAt(i);
    hash  = ((hash << 5) - hash) + chr;
    hash |= 0; // Convert to 32bit integer
  }
  return hash;
};

Array.prototype.isUnique = function(obj){
 if (this.length <= 0)
   return true;
    
 let hash = JSON.stringify(obj).hashCode();
 return !this.some(function(e){
  return JSON.stringify(e).hashCode() === hash;
 });
}

let test = [{id: 11, type: "sell", quantity: 11, price: 155},
{id: 11, type: "sell", quantity: 11, price: 155},
{id: 11, type: "sell", quantity: 11, price: 155},
{id: 12, type: "buy", quantity: 3, price: 189},
{id: 13, type: "buy", quantity: 4, price: 189},
{id: 14, type: "buy", quantity: 2, price: 189},
{id: 14, type: "buy", quantity: 2, price: 189}];


let out = [];

test.forEach(function(item){
 if (out.isUnique(item))
  out.push(item);
});

console.log(out);

While this answer solves the problem, it's not really *that* simple and also not really that performant. — comu, Sep 05 '18 at 19:27

comu · Answer 3 · 2018-09-05T19:29:58.243

This question has been downvoted, but I don't think I've ever really seen a succinct answer on how to do this with object keys. If you don't need IE11, this answer should work. It uses the ES6 findIndex function.

function removeDuplicatesFromArray(arr){
  let unique_array = arr.filter(function(elem, index, self) {
      let firstOccurence = self.findIndex(elem => elem.id == self[index].id)
      return firstOccurence == index;
  });

  return unique_array
}

var duplicatesArr = [{id: 11, type: "sell", quantity: 11, price: 155},
{id: 11, type: "sell", quantity: 11, price: 155},
{id: 11, type: "sell", quantity: 11, price: 155},
{id: 12, type: "buy", quantity: 3, price: 189},
{id: 13, type: "buy", quantity: 4, price: 189},
{id: 14, type: "buy", quantity: 2, price: 189},
{id: 14, type: "buy", quantity: 2, price: 189}]


console.log(removeDuplicatesFromArray(duplicatesArr));

Also:

I would recommend checking out some of the other built in Array functions available. Most of them are ES2015, and therefore compatible with IE11. My specific answer used Array.prototype.filter, but you could have also used map, among others.

How to remove duplicates items from the list in javascript

3 Answers3