10

I am working with Geofire and Firebase on Angular 6 to store locations and unfortunately it's storing a lot of duplicates this is an example (console logging my variable currentHits):

0: {location: Array(2), distance: "48.84", url: "assets/imgs/fix.png"}
1: {location: Array(2), distance: "48.84", url: "assets/imgs/fix.png"}
2: {location: Array(2), distance: "48.84", url: "assets/imgs/fix.png"}
3: {location: Array(2), distance: "48.85", url: "assets/imgs/free.png"}
4: {location: Array(2), distance: "48.85", url: "assets/imgs/free.png"}
5: {location: Array(2), distance: "48.85", url: "assets/imgs/free.png"}
6: {location: Array(2), distance: "48.87", url: "assets/imgs/low.png"}
7: {location: Array(2), distance: "48.87", url: "assets/imgs/low.png"}
8: {location: Array(2), distance: "48.87", url: "assets/imgs/low.png"}

Location basically is an array of latitude and longitude used to calculate distance, in id 0, 1 and 2 its the same coordinates, and 3,4 and 5 are also the same, ...

This is what I want to get:

0: {location: Array(2), distance: "48.84", url: "assets/imgs/fix.png"}
1: {location: Array(2), distance: "48.85", url: "assets/imgs/free.png"}
2: {location: Array(2), distance: "48.87", url: "assets/imgs/low.png"}

(Optional) this is how It stores these locations:

  ...
  hits = new BehaviorSubject([])

  ...
  queryHits(...){
 ....
 let hit = {
          location: location,
          distance: distance.toFixed(2),
          url:img
        }

        let currentHits = this.hits.value
        currentHits.push(hit)
        this.hits.next(currentHits)
....
}

It's true that this question has probably already been asked and I have been digging through all the similar questions and found these functions:

1. RemoveDuplicates()

function removeDuplicates(arr){
    let unique_array = []
    for(let i = 0;i < arr.length; i++){
        if(unique_array.indexOf(arr[i]) == -1){
            unique_array.push(arr[i])
        }
    }
    return unique_array
}

var newlist = removeDuplicates(list)

It didn't work I get the same list with duplicates.

2. arrUnique:

function arrUnique(arr) {
    var cleaned = [];
    arr.forEach(function(itm) {
        var unique = true;
        cleaned.forEach(function(itm2) {
            if (_.isEqual(itm, itm2)) unique = false;
        });
        if (unique)  cleaned.push(itm);
    });
    return cleaned;
}

var newlist= arrUnique(list);

Also, it didn't work..

3. onlyUnique

  onlyUnique(value, index, self) { 
    return self.indexOf(value) === index;
  }

var newlist = list.filter(onlyUnique)

Unfortunately it didn't work...

These are some of the answers given to similar problem to remove duplicates from an array and none of them worked. I don't understand why they won't work for my type of array, If anyone has an idea or knows why would be very helpful.

Programmer Man
  • 1,314
  • 1
  • 9
  • 29
  • Your approaches didn't work because all objects, even if it shares all the same key/values, are different instances (unless Singleton which this isn't) – Mark Sep 24 '18 at 13:33
  • Can you share sample JSON. That would be better for us – Rajesh Sep 24 '18 at 13:34
  • would it be also a solution for you to not use incremental id's (0, 1, 2, 3,...) as the prop name but use object related id's ? This will ensure you, that you have no duplicated records. – ylerjen Sep 24 '18 at 13:34
  • 1
    you can't compare objects this way : `{} === {}` return `false` – Martial Sep 24 '18 at 13:36
  • This is not a duplicate and you are right @RobG I changed to Geofire objects – Programmer Man Sep 24 '18 at 13:38
  • "*Location basically is an array of latitude and longitude…*" your example doesn't show any latitude or longitude values, what are you actually trying to compare? – RobG Sep 24 '18 at 13:44
  • As I said location is an array for id 0,1,2, is the same thats why I get same distance because its calculated from the coordinates – Programmer Man Sep 24 '18 at 14:00
  • Sure, but do you want to detect duplicate lat/long, duplicate distance, or where all properties and values are the same? – RobG Sep 24 '18 at 20:23
  • Please rob this question is not a duplicate I have made a research before making this question and none worked for me. – Programmer Man Sep 24 '18 at 20:33

7 Answers7

3

You could use a set to store and check for duplicate values.

const removeDuplicates = arr => {
    let matches = new Set();
    return arr.filter(elem => {
        const {distance} = elem;
        if(matches.has(distance)){
            return false;
        } else {
            matches.add(distance);
            return true;
        }
    })   
}

Bear in mind that using this approach you may remove results where the distance is the same but the co-ordinates differ. If that causes an issue for you then you'd need to also check against the lat/lng pair.

Adam
  • 1,724
  • 13
  • 16
3

Problem here is comparing Objects. Two objects are never equal unless both are referencing to same Object.

Example:

{} === {} // false

// Two objects are equal only if they are referencing to same object
var a = {};
a === a; // true

It is clear from your problem that you are facing the first case. Among the solutions you tested Solution 1 and Solution 3 are failing because of this reason as indexOf does === comparision.

But Solution 2 should have worked on your example as it does a deep comparision as explained here. https://lodash.com/docs#isEqual.

PS: It might be a simple typo i have observed in Solution 2 cleaned.,push(itm);, there is an extra comma. Hoping that is not the case I am moving ahead

So i guess the issue is inside your location array, if you can give the contents of location array we should be able to provide better solution. Or as others suggested you can filter based on a single key of the object like id or distance, instead of comparing the whole object

AvcS
  • 2,263
  • 11
  • 18
  • 1
    Yes, though solution 2 could be written a little simpler using `if (!_.isEqual(itm, itm2)) cleaned.push(itm);`. ;-) – RobG Sep 24 '18 at 13:52
1

You could always check before you add the hits to make sure there are no repeats.

edit: You cannot compare objects unless they have the same reference object. So, you could compare objects by a unique ID

use rxjs filter() this will return an array

// store history of objs for comparison
addedObjs = [];
this.hits.pipe(filter(obj => {
    // check if id is an index of the previous objs
    if (addObjs.indexOf(obj.id) === -1) {
        this.addedObjs.push(obj.id)
        return obj
    });

here is the working stackblitz using some of your code

FussinHussin
  • 1,718
  • 4
  • 19
  • 39
  • This approach is soo gooad but I am getting error ts] The 'this' context of type 'void' is not assignable to method's 'this' of type 'Observable<{}> – Programmer Man Sep 24 '18 at 13:41
  • the list I showd on my question actually can be logged from console.log(this.hits.value) or console.log(currentHits) – Programmer Man Sep 24 '18 at 13:42
  • @ProgrammerMan it's probably a bad import from rxjs. If you are using rxjs 6, import { distinctUntilChanged } from 'rxjs/operators' – FussinHussin Sep 24 '18 at 13:46
  • Thank you for checking I have changed the import as you said and now getting this Argument of type '(value: any[]) => any[]' is not assignable to parameter of type '(x: any[], y: any[]) => boolean'. Type 'any[]' is not assignable to type 'boolean'. – Programmer Man Sep 24 '18 at 13:48
  • @ProgrammerMan my apologies, actually you don't need to pass value in the operator, i'll update the answer – FussinHussin Sep 24 '18 at 13:50
  • still getting errors the thing is this.hits is BehaviorSubject([]) Object – Programmer Man Sep 24 '18 at 14:13
  • hmm, you would have to post what you are doing for more help, or add me in a chat. because you can use distinctUntilChanged on a behavior subject https://stackoverflow.com/questions/39667133/behavioursubjects-distinctuntilchanged-is-not-a-function – FussinHussin Sep 24 '18 at 14:20
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/180676/discussion-between-fussinhussin-and-programmer-man). – FussinHussin Sep 24 '18 at 14:26
1

uniqWith https://lodash.com/docs/#uniqWith can be used to specify the method to compare by :

var arr = [ { location: [1, 2], distance: "48.84", url: "assets/imgs/fix.png" },
            { location: [1, 2], distance: "48.84", url: "assets/imgs/fix.png" },
            { location: [1, 2], distance: "48.84", url: "assets/imgs/fix.png" },
            { location: [3, 4], distance: "48.85", url: "assets/imgs/free.png"},
            { location: [3, 4], distance: "48.85", url: "assets/imgs/free.png"},
            { location: [3, 4], distance: "48.85", url: "assets/imgs/free.png"},
            { location: [5, 6], distance: "48.87", url: "assets/imgs/low.png" },
            { location: [5, 6], distance: "48.87", url: "assets/imgs/low.png" },
            { location: [5, 6], distance: "48.87", url: "assets/imgs/low.png" } ]
          
var result = _.uniqWith(arr, (a, b) => _.isEqual(a.location, b.location));

console.log( JSON.stringify({...result}).replace(/},/g, '},\n ') );
<script src="https://cdn.jsdelivr.net/npm/lodash@4.17.11/lodash.min.js"></script>
Slai
  • 22,144
  • 5
  • 45
  • 53
  • This won't work since it doesn't take in consideration the unique id each object has. – Programmer Man Sep 24 '18 at 13:45
  • 1
    Also, you can't guarantee the order of object properties, so can't guarantee that the JSON.stringify will be a useful comparison. – RobG Sep 24 '18 at 13:46
1

You can use following approach:

Idea:

  • You can create your own data structure and have a hashMap to save values.
  • Since you have location data, you can use longitude|latitude as your key name as it will be unique.
  • Then expose some functions say, add that will check if value exists, override else add.
  • Also create a property, say value that would return the list of locations.

Note: Above behavior can be achieved using Set as well. If you cannot use ES6 features, then this is one way that is extensible and easy.

function MyList() {
  var locations = {};

  this.add = function(value) {
    var key = value.location.join('|');
    locations[key] = value;
  }

  Object.defineProperty(this, 'value', {
    get: function() {
      return Object.keys(locations).map(function(key) {return locations[key] })
    }
  })
}

var locations = new MyList();

locations.add({location: [123.12, 456.23], name: 'test 1' });
locations.add({location: [123.16, 451.23], name: 'test 2' });
locations.add({location: [123.12, 456.23], name: 'test 1' });
locations.add({location: [100.12, 456.23], name: 'test 3' });
locations.add({location: [123.12, 456.23], name: 'test 1' });
locations.add({location: [123.12, 400.23], name: 'test 4' });

console.log(locations.value)

Typescript version for more readability:

interface ILocation {
  location: Array<number>
  [key: string]: any;
}

interface IList {
  [key: string]: ILocation
}

class MyList {
  private locations: IList = {};
  
  public add(value: ILocation) {
    const key: string = value.location.join('|');
    this.locations[key] = value;
  }
  
  public get value(): Array<ILocation> {
    return Object.keys(locations).map(function(key) {return locations[key] })
  }
}
Community
  • 1
  • 1
Rajesh
  • 24,354
  • 5
  • 48
  • 79
0

Perhaps you would want to use a library such as lodash which has a wide set of functions regarding all types of collections.

let newArr = _.uniqWith(myArr, _.isEqual);

uniqWith with the help of isEqual can get what you want.

Here is fiddle to that solution

malek yahyaoui
  • 379
  • 3
  • 17
0

I think it is your comparison that may not be working correctly. You can try this:

var uniqueHits = currentHits.reduce((acc, curr) => { 
    if (acc.length === 0) {
        acc.push(curr);
    } else if (!acc.some((item) => 
        item.location[0] === curr.location[0]
        && item.location[1] === curr.location[1]
        && item.distance === curr.distance
        && item.url === curr.url)) {
        acc.push(curr);
    }
    return accumulator;
}, []);;
Frank Fajardo
  • 7,034
  • 1
  • 29
  • 47