0

I have an array of objects with about 1500 elements, I am trying to make a new array removing the elements that have a duplicate unique property. But for some reason when I run the function it stops at the first 100 elements of the array. How can I get it to loop through the whole array.

  const result = Array.from(new Set(DATA.map((a) => a.Numbers))).map(
    (Numbers) => {
      return DATA.find((a) => a.Numbers === Numbers);
    }
  );
Yevhen Horbunkov
  • 14,965
  • 3
  • 20
  • 42
Shiva
  • 1
  • 1
  • 1
    Just a side note. Your dedupe implementation seems to be not quite optimal from performance point of view. – Yevhen Horbunkov Sep 03 '21 at 15:31
  • Thanks for the heads up, still a newbie here will try other implementations. – Shiva Sep 03 '21 at 15:45
  • 1
    Are you sure there are more than 100 different `Numbers` properties? I don't see any reason why this code would stop short. – Barmar Sep 03 '21 at 15:46
  • Yea I am positive I just checked my DATA file even the last element line has the property Numbers. I can't wrap my head around it. – Shiva Sep 03 '21 at 15:50
  • @Shiva *"even the last element line has the property Numbers"* That doesn't answer the question of Barmar. The question is if you have more than 100 **different** `Numbers`. It might just be that your `DATA` array only contains 100 unique `Numbers` properties. eg. `DATA = [{ Numbers: 1 }, { Numbers: 2 }, { Numbers: 1 }]` only contains 2 unique `Numbers` properties, even though there are 3 objects. – 3limin4t0r Sep 03 '21 at 15:52
  • Perhaps you could put your data in a pastebin so we can take a look at it. – Andy Sep 03 '21 at 15:54
  • https://pastebin.com/knT3eYyx maybe I am mistaken on the outcome of this, I was expecting just any duplicates to be removed like if there were two elements with Numbers: "14" to only return one of them – Shiva Sep 03 '21 at 16:00
  • It is exactly doing that. Looking at your pastebin `Numbers` seems to be a value between `00` and `99` (inclusive). So I'm not exactly sure why you are expecting more than 100 results. – 3limin4t0r Sep 03 '21 at 16:14
  • And, just in case, your solution is perfectly valid, except for certain [performance](https://jsbench.me/rtkt4jfs64/4) concerns. – Yevhen Horbunkov Sep 03 '21 at 16:19
  • Yea I see that, is there away I can make it to do only the first 80 unique elements of the array. That is my ultimate goal. I want the first 80 unique elements. And then for the function to start over and get the next 80 unique elements. – Shiva Sep 03 '21 at 16:24
  • Why don't you just get all the unique results, then split it up into chunks of 80 afterward? – Barmar Sep 03 '21 at 16:29
  • 1
    That's a completely different question than this. – Barmar Sep 03 '21 at 16:30
  • Yea thanks will try around I am not sure why I was expecting something different here. Thanks for the tip – Shiva Sep 03 '21 at 16:36
  • **Beware** that the data from your pastebin has all its zero-leading-digit `Numbers` stored as **strings**: e.g.: `'09'` But its non-zero-leading-digit `Numbers` stored as actual JavaCcript **Numbers**: e.g.: `92` – Wyck Sep 03 '21 at 17:17
  • Yea trying to figure out what would be the best route to take. The data needs to have a leading 0 if its a single digit, should I convert all to string? – Shiva Sep 04 '21 at 01:53

4 Answers4

0

You're really complicating matters. You're mapping twice, converting the result into a set, and then creating a new array from that set.

It would be much simpler (and more readable) to use a simple loop, and keep a record of the numbers in the objects. If a number already exists splice the object from the array.

This method won't create a new array - you're modifying the existing one - but it will work.

const arr = [{ number: 1 },{ number: 2 },{ number: 3 },{ number: 4 },{ number: 1 },{ number: 4 }];

const numbers = new Set();

for (let i = arr.length - 1; i >= 0 ; i--) {
  const { number } = arr[i];
  if (numbers.has(number)) arr.splice(i, 1);
  numbers.add(number);
}

console.log(arr);
Andy
  • 61,948
  • 13
  • 68
  • 95
  • `tally` should be a `Set` to make testing inclusion efficient. – Barmar Sep 03 '21 at 15:48
  • Thanks for the heads-up, @Barmar. How much more efficient is it? – Andy Sep 03 '21 at 15:52
  • 1
    Set searching is O(1), list searching is O(n). – Barmar Sep 03 '21 at 15:53
  • Ah, ok. Thanks. I've never really understood O numbers. – Andy Sep 03 '21 at 15:55
  • 1
    Those are worst case scenarios. If you use an array and do `numbers.includes(number)`, in the worst case scenario `includes` iterates the whole array O(n). This happens when the value you are searching is the last element or not present. Sets are indexed (similar to object properties) so `numbers.has(number)` would "instantly" O(1) (it still has to navigate through some sort of search tree) know if the value is or is not present without iterating the full collection. – 3limin4t0r Sep 03 '21 at 16:03
  • That makes perfect sense. Thanks @3limin4t0r. – Andy Sep 03 '21 at 16:13
0

Create an object that uses the Numbers property as the keys. Since object keys must be unique, this will remove duplicates. Then get the object values to convert back to an array.

const DATA = [{ Numbers: 1 },{ Numbers: 2 },{ Numbers: 3 },{ Numbers: 4 },{ Numbers: 1 },{ Numbers: 4 }];
const result = Object.values(Object.fromEntries(DATA.map(a => [a.Numbers, a])));
console.log(result)
Barmar
  • 741,623
  • 53
  • 500
  • 612
  • Note that this solution takes the last object encountered rather than the first one (on duplicate). This might be perfectly fine depending on the scenario you're in. – 3limin4t0r Sep 03 '21 at 16:22
0

Since no Map-based answers yet (and I believe, Map suits the purpose the best from performance standpoint), I'll post mine:

const src = [{key: 'a', value: 1}, {key: 'c', value: 3}, {key: 'b', value: 2}, {key: 'a', value: 1}, {key: 'c', value: 3}]

const dedupe = (arr, keyProp) => [
    ...arr
        .reduce((acc, obj) => 
            (acc.set(obj[keyProp], obj), acc), new Map)
        .values()
]

const result = dedupe(src, 'key')

console.log(result)
.as-console-wrapper{min-height:100%;}
Yevhen Horbunkov
  • 14,965
  • 3
  • 20
  • 42
  • Note that this solution takes the last object encountered rather than the first one (on duplicate). This might be perfectly fine depending on the scenario you're in. – 3limin4t0r Sep 03 '21 at 16:31
  • @3limin4t0r : well, yeah, it is based on assumption that items with the same value of target prop have ***identical*** rest of the props. Otherwise, none of the posted solutions are valid, since OP's and Andy's solution will pick only first match (by target prop, ignoring the rest), while Barmar's and mine will return the last occurrence. – Yevhen Horbunkov Sep 03 '21 at 16:53
0

The idiom for making an array of distinct objects (also described in this answer) goes like this:

const distinct = DATA.filter((obj, idx) => 
  idx === data.findIndex(a => a.Numbers === obj.Numbers));

This filters the input array by selecting all items that, when linearly searched for, return the same index they already have. Thus selecting the first of each such object with the given criteria.

Note: that some of your Numbers were strings and some were actual numbers. (Those with a leading 0 were stored as strings, like '02'.) You could use the less strict == instead of === if you need to deal with situations where the same value may be stored in both string and number format. e.g.: a.Numbers == obj.Numbers.

Wyck
  • 10,311
  • 6
  • 39
  • 60
  • 2
    Despite being quite straightforward, hence easy to comprehend by novice, your solution clearly implements O(n²)-time algorithm which will be [terribly slow](https://jsbench.me/rtkt4jfs64/5), should your input array be large enough. That may be totally fine for OP's use case, however, is worth mentioning. – Yevhen Horbunkov Sep 03 '21 at 17:46