Ionic 2 - How to put a large JSON data in variable and remove the duplicates without lag?

Question

I have a JSON file that contain more than 40K (40000) lines.

I used this code to get the data of json and put the received data in variable:

this.http.get(url).map(res => res.json()).subscribe(data => {
    ...
    this.users=data.users;
    ...
    this.images=data.images;
    this.removeDupicates();
    ...
});

My problem: when the data of JSON is loaded and try to put the data in variable and try to remove the duplicates, the application take long time to be ready.

So is there any solutions that speed up this process or fix it ?

EDIT:

Here is the code of function "removeDuplicates"

removeDupicates(){
   for(let i=0; i<this.images.length; i++){
      for(let j=0; j<this.images.length; j++){
         if(this.images[i].img == this.images[j].img && i!=j){
            this.images.splice(i,1);
         }
      }
   }
}

score 3 · Accepted Answer · answered May 23 '18 at 07:37

So as found in this answer and modified to your case:

Array.from(new Set(
[
  {"img":"1"},
  {"img":"2"},
  {"img":"1"},
  {"img":"3"}
].map((itemInArray) => itemInArray.img)));

More about Array.from & Set

The output will be ["1,"2","3"]

Now, if you want this to happen asynchronous, thus not having to wait for the data to be loaded before continuing with the app, you can wrap it in a setTimeout

So:

removeDuplicates() {
   setTimeout(() => {
     this.images = Array.from(new Set(
          this.images.map(image => image.img)
     ));
   }, 0); // execute timeout function immediately, fakes async
}

Note that you won't have a list like : [{img: 'myimage'}, ....] but you now have a list ['myimage', 'mysecondimage', ...]

Ionic 2 - How to put a large JSON data in variable and remove the duplicates without lag?

1 Answers1