NodeJS: Extracting duplicates from Array

Question

I'd need your help... I'm sorry if the question has already been asked but I can't seem to find a suitable answer for my problem: I'm trying to extract (not remove) the list of the duplicates from my array.

ultimately, the main objective is to keep only one of the duplicated objects (in the array) with the higher profit...

Here's a simple example of my array:

var arr = [
  {
    mkBase: "test",
    mkComp: "test1",
    invest: { profit: 10 },
    availability: true,
    option: 1
  },
  {
    mkBase: "test",
    mkComp: "test1",
    invest: { profit: 15 },
    availability: false,
    option: 2
  },
  {
    mkBase: "test1",
    mkComp: "test",
    invest: { profit: 8 },
    availability: true,
    option: 3
  },
  {
    mkBase: "test2",
    mkComp: "test",
    invest: { profit: 6 },
    availability: true,
    option: 4
  },
  {
    mkBase: "test",
    mkComp: "test2",
    invest: { profit: 6 },
    availability: true,
    option: 5
  },
  {
    mkBase: "test",
    mkComp: "test3",
    invest: { profit: 7 },
    availability: true,
    option: 6
  },
  {
    mkBase: "test",
    mkComp: "test3",
    invest: { profit: 10 },
    availability: true,
    option: 7
  },
  {
    mkBase: "test3",
    mkComp: "test4",
    invest: { profit: 10 },
    availability: true,
    option: 8
  }
];

And I managed to extract a list of almost all duplicates using:

for (var i = 0; i < arr.length; i++) {
  if (_.uniqBy(arr, "mkBase").indexOf(arr[i]) == -1) {
    console.log("[SAME BASE]: " + JSON.stringify(arr[i], null, 2));
  } else if (_.uniqBy(arr, "mkComp").indexOf(arr[i]) == -1) {
    console.log("[SAME COMP]: " + JSON.stringify(arr[i], null, 2));
  }
}

And here's the result:

[SAME BASE]: {
  "mkBase": "test",
  "mkComp": "test1",
  "invest": {
    "profit": 15
  },
  "availability": false,
  "option": 2
}
[SAME COMP]: {
  "mkBase": "test2",
  "mkComp": "test",
  "invest": {
    "profit": 6
  },
  "availability": true,
  "option": 4
}
[SAME BASE]: {
  "mkBase": "test",
  "mkComp": "test2",
  "invest": {
    "profit": 6
  },
  "availability": true,
  "option": 5
}
[SAME BASE]: {
  "mkBase": "test",
  "mkComp": "test3",
  "invest": {
    "profit": 7
  },
  "availability": true,
  "option": 6
}
[SAME BASE]: {
  "mkBase": "test",
  "mkComp": "test3",
  "invest": {
    "profit": 10
  },
  "availability": true,
  "option": 7
}

The Lodash method (_.uniqBy) is keeping one of the duplicates in the main Array, and, in order to ultimately get the best (_.maxBy(arr, 'profit')) of the duplicates, I'd need it with the other duplicates.

I'm not sure I'm very clear, but if you need any clarification please let me know!

Thanks in advance to you all!

********** EDIT ************* As suggested by stasovlas you'll find below the expected result and why the other objects in the array were removed:

var result = [
  {
    mkBase: "test",
    mkComp: "test1",
    invest: { profit: 15 },
    availability: false,
    option: 2
  },
  {
    mkBase: "test1",
    mkComp: "test",
    invest: { profit: 8 },
    availability: true,
    option: 3
  },
  {
    mkBase: "test3",
    mkComp: "test4",
    invest: { profit: 10 },
    availability: true,
    option: 8
  }
];

var removed = [
  //Reason: Same Base **and** Comp mk as option 2 && Profit is too low versus option 2
  {
    mkBase: "test",
    mkComp: "test1",
    invest: { profit: 10 },
    availability: true,
    option: 1
  },
  //Reason: Same Comp mk as option 3 && Profit is too low versus option 3
  {
    mkBase: "test2",
    mkComp: "test",
    invest: { profit: 6 },
    availability: true,
    option: 4
    //Reason: Same Base mk as option 2 && Profit is too low versus option 2
  },
  {
    mkBase: "test",
    mkComp: "test2",
    invest: { profit: 6 },
    availability: true,
    option: 5
  },
  //Reason: Same Base mk as option 2 && Profit is too low versus option 2 
  {
    mkBase: "test",
    mkComp: "test3",
    invest: { profit: 7 },
    availability: true,
    option: 6
  },
  //Reason: Same Base mk as option 2 && Profit is too low versus option 2 
  {
    mkBase: "test",
    mkComp: "test3",
    invest: { profit: 10 },
    availability: true,
    option: 7
  }
];

Do you need the duplicate items anytime in your app ? You might want to take a look at this thread about data structure: "Don't use an array when you actually need a set" : https://stackoverflow.com/questions/40055764/setting-arrays-in-firebase-using-firebase-console#answer-40055996 I know the context is Firebase but you might find an answer there — t3__rry, Jan 18 '18 at 16:02
Thanks I'll be looking into it, but I'm not familiar at all with Firebase. And to answer your question, no I won't need it afterwards... Once I've selected the objects (from the duplicates) that has the highest profit value, I basically throw the others away. — Martin Carre, Jan 18 '18 at 16:46
Glad if it helps. Firebase is a different topic but the answer it provides here might be a good reflexion to have on the data structure (I mean, it made me think differently as well on data structure and habits we all have) Take a look at this answer which is what you're looking for btw: https://stackoverflow.com/questions/16747798/delete-duplicate-elements-from-an-array#answer-16747921 — t3__rry, Jan 18 '18 at 16:49

score 1 · Answer 1 · answered Jan 18 '18 at 20:06

1

var sameBase = {}, sameComp = {};

arr.forEach(item => {
    let existingBase = sameBase[item.mkBase];

    if ( ( existingBase === undefined ) ||  ( existingBase.invest.profit < item.invest.profit ) ) {
            sameBase[item.mkBase] = item;  
    } 

    existingComp = sameComp[item.mkComp];
    if ( ( existingComp === undefined ) ||  ( existingComp.invest.profit < item.invest.profit ) ) {
        sameComp[item.mkComp] = item;  
    }
});

var sameBaseArr = toArr(sameBase);
var sameCompArr = toArr(sameComp);

console.log("SAME BASE: " + JSON.stringify(sameBaseArr, true));
console.log("SAME COMP: " + JSON.stringify(sameCompArr, true));

function toArr(map) {
    let arr = [];
    for (var key in map) {
        arr.push(map[key]);
    }   
    return arr;     
}

answered Jan 18 '18 at 20:06

kofman

108
6

Hey Kofman! I think you got something there... I'd need to combine your 2 functions though! Let me check the others' answers and try to combine and I'll come back to you! Thanks! – Martin Carre Jan 19 '18 at 07:46
1

Based on what criteria you want to combine? Do you need to just concat the two arrays? Or, deduplicate further based on some equality criteria? E.g. obj1 equals obj2 iff obj1.mkBase equals obj2.mkComp? - something like that? Need more info... – kofman Jan 19 '18 at 15:04
No problem at all Kofman and thanks for the follow-up... I need to remove the duplicates whether it's the mkBase or the mkComp that are duplicated ie. I need to remain with only a unique mkBase object and a unique mkComp object and the one that I keep needs to be the one with the highest profit. Plus, if in the Array (`arr`) there are non-duplicates objects (no other mkBase or mkComp (as it is the case with option 8) I need to keep them. I hope this clarifies things! Thanks again! :P – Martin Carre Jan 19 '18 at 15:35

stasovlas · Accepted Answer · 2018-01-19T08:56:50.023

1

i'm not sure in my question understanding, but here my solution:

const res = _.reduce(arr, (result, item) => {
    const same = _.find(result, r => _.some([
        r.mkBase === item.mkBase,
        r.mkComp === item.mkComp
    ])); // find same already added item

    if (same === undefined) {
        return _.concat(result, item); // just push item   
    }

    if (same.invest.profit >= item.invest.profit) {
        return result; // do nothing if profit is less then already added
    }

    return _.chain(result) // remove item with smaller profit and push item with higher profit
        .reject({ mkBase: same.mkBase, mkComp: same.mkComp })
        .concat(item)
        .value();
}, []);

edited Jan 19 '18 at 08:56

answered Jan 19 '18 at 07:12

stasovlas

7,136
2
28
29

Hey @stasovlas! Thanks for your answer but unfortunately, **ultimately** I'm looking to get rid of all the duplicates keeping only the highest profit one. For instance: I can't get as a base: "Test" with "test2" as comp and another one with base 'test3' and 'test2' as comp. Just as I can't get "test" as base with "test2" as comp and again base "test and comp "test3" for instance ... I'm sorry this is not very clear I'm afraid! But let me know how I can help clarifying things up... – Martin Carre Jan 19 '18 at 07:41
1

@Ardzii can you add expected result in your question? – stasovlas Jan 19 '18 at 08:11
Done! Thanks stasovlas for suggesting this! :) – Martin Carre Jan 19 '18 at 08:20
That seemed to have done the trick! Thanks mate! I need to get my head around your answer to apply it now! I'll keep you posted! Thanks again! – Martin Carre Jan 19 '18 at 09:08
A quick question: I'm having a really hard time understanding what is it that the lodash method `_.reduce()` does... I've been there: https://lodash.com/docs/4.17.4#reduce and can't seem to wrap my head around what happens and how... :( sorry to be a pain in the *** but I'd be really grateful if you could help me understand. – Martin Carre Jan 19 '18 at 12:17
1

@Ardzii huh, my english is too bad for explanation, I hope this article helps you https://medium.freecodecamp.org/reduce-f47a7da511a9 – stasovlas Jan 19 '18 at 12:23

Lyçann H. · Answer 3 · 2018-01-18T16:59:38.847

Try this (not tested, but the logic can help):

var mkBases = [], mkComps = [];

for (var i = 0; i < arr.length; i++) {
    var actualBase = arr[i].mkBase;
    var actualComp = arr[i].mkComp;

    if (typeof mkBases[actualBase] !== 'undefined') {
        mkBases[actualBase] = {getAllIndexes(arr, actualBase)}
    }

    if (typeof mkComps[actualComp] !== 'undefined') {
        mkComps[actualComp] = {getAllIndexes(arr, actualComp)}
    }
}

function getAllIndexes(arr, val) {
    var indexes = [], i = -1;
    while ((i = arr.indexOf(val, i+1)) != -1){
        indexes.push(i);
    }
    return indexes;
}

Now you can iterate over mkBases and mkComp to get your duplicates.

NodeJS: Extracting duplicates from Array

3 Answers3