Javascript: Determine and return duplicates objects from array of objects based in multiple keys

Question

I have an array of objects:

    var data  = [{"monitor":"TFT",
                  "manufacturer":"MONCORP",
                  "monID":"1234",
                  "Delivery Way":"DELIVERY",
                  "BarCode Text":"Test",
                  "BarCode Id":"D9",
                  "Status":"OK"},               

                 {"monitor":"LCD",
                  "manufacturer":"MONCORP",
                  "monID":"",
                  "Delivery Way":"PICKUP",
                  "BarCode Text":"Dummy Text",
                  "BarCode Id":"P2",
                  "Status":"OK"},

                 {"monitor":"TFT",
                  "manufacturer":"MONCORP",
                  "monID":"1234",
                  "Delivery Way":"DELIVERY",
                  "BarCode Text":"ONLY TEST",
                  "BarCode Id":"D9",
                  "Status":"OK"}, 

                 {"monitor":"LCD",
                  "manufacturer":"MONCORP",
                  "monID":"1234",
                  "Delivery Way":"DELIVERY",
                  "BarCode Text":"FOR TESTING PURPOSE",
                  "BarCode Id":"D9",
                  "Status":"OK"},

                  {"monitor":"TFT",
                   "manufacturer":"MONCORP",
                   "monID":"",
                   "Delivery Way":"PICKUP",
                   "BarCode Text":"DUMMIEST TEXT",
                   "BarCode Id":"P7",
                   "Status":"OK"}];

So I want to take only the objects that are duplicated, BUT I want to distinguish them according to values of keys: monitor, manufacturer, monID, Delivery Way, BarCode Id, Status.

The expected result is:

    expected = [{"monitor":"TFT",
                 "manufacturer":"MONCORP",
                 "monID":"1234",
                 "Delivery Way":"DELIVERY",
                 "BarCode Text":"Test",
                 "BarCode Id":"D9",
                 "Status":"OK"},

                {"monitor":"TFT",
                 "manufacturer":"MONCORP",
                 "monID":"1234",
                 "Delivery Way":"DELIVERY",
                 "BarCode Text":"ONLY TEST",
                 "BarCode Id":"D9",
                 "Status":"OK"}]

by 'I want to distinguish them according to values of keys:..' do you mean you want to ignore the differences for 'BarCode Text'? (or any other key not in the list?) — Kaddath, Jun 29 '17 at 09:01
Yes exactly i am only interested for the list: monitor, manufacturer, monID, Delivery Way, BarCode Id, Status. — Alex Botzis, Jun 29 '17 at 09:03

emanek · Answer 1 · 2017-06-29T11:33:27.510

1

It might be a good idea to do this on the database level (e.g. some sort of GROUP BY operation).

In JavaScript, you can iterate through the array and create a hash for each record, which should be made up of the fields that you want to be unique. This hash can be then used as a map key, i.e. inserting these record to a map with such key will eliminate the duplicates.

Example:

var map = {};
for (var i = 0; i < data.length; i++) {
    var key = data[i].monitor + "#" + data[i].monID + "#" + data[i].manufacturer + ... ;
    map[key] = data[i];
}

The map object will at the end contain only unique keys mapped to the last item with such key.

Note that the key is just a string concatenated of the fields you want to be unique. It could not work correctly if your fields contain the character #, or if they are not type string. If you want to go this way, I would suggest to compute a hash code from the string

To identify the duplicates, you can check in every step of the iteration whether the key is already in the map:

if (map[key]) {
   // This record is a duplicate
}

In order to group together the duplicate records, you can construct a map mapping key -> array of duplicates. This could be done like this (showing just the inside of the loop)

var key = ...
if (!map[key]) {
    // First time we see this key, let's add it to the map
    map[key] = [ data[i] ]; // Map the key to a new array containing the current record
} else {
    // Duplicate; just add this record to the existing 
    // list of records with the same key
    map[key].push(data[i]);
}

edited Jun 29 '17 at 11:33

answered Jun 29 '17 at 09:04

emanek

411
2
8

this method works well and is commonly used. Don't forget to adapt it a little if the values in your data can be something else than strings. In case of objects or arrays, you can use `JSON.stringify`. In this case, you should use a hash code as suggested to avoid too long strings – Kaddath Jun 29 '17 at 09:18
Can we somehow get rid of these lines (these that i mentioned it expected result) ? – Alex Botzis Jun 29 '17 at 09:25
Sorry what do you mean by get rid off these lines...? – emanek Jun 29 '17 at 09:32
To reject those lines (actually to store them in a log file) who have in common values of keys: monitor, manufacturer, monID, Delivery Way, BarCode Id, Status. – Alex Botzis Jun 29 '17 at 09:34
@AlexBotzis yes, in the line beginning by `var key = data[i].monitor`. You explicitly specify here the fields you will use to compose your key. Just don't put here the fields you want to ignore – Kaddath Jun 29 '17 at 09:35
OK I understood this, but after how can i reject those lines based on the key? – Alex Botzis Jun 29 '17 at 09:39
@AlexBotzis if I understand correctly, you want to identify the duplicates... I edited the answer to account for this; just check in every step if the key is already there... – emanek Jun 29 '17 at 09:39
@emanek I tried to understand your concept and i believe that is correct, but i cannot log the duplicates..Can you help me with that ?? Thank you ... – Alex Botzis Jun 29 '17 at 11:20
@AlexBotzis I extended the answer to show you how to possibly construct list of duplicates, not sure how exactly you want to 'log' them... please accept the answer if it solved your problem, thanks – emanek Jun 29 '17 at 11:34
Yes it works..Thank you very much for the explanation – Alex Botzis Jun 29 '17 at 12:31

Javascript: Determine and return duplicates objects from array of objects based in multiple keys

1 Answers1