Using AngularFire2 to check if large amount of data exists in Firebase database

Question

Best Practice Question

The situation:

Using Angularfir2 I am trying to find the best way to check which, if any, of a list of items that I have stored locally exist within my Firebase database. This is considering that I have a very large amount of data in my Firebase database (in this example 100k + entries but could be well over 1M) and a good deal of entries locally (in this example 200 + however it could be over 1000)

So here is an example of the Firebase database:

"project-1234": [
  {
    "animals" : {
      "1" : {
        "animal" : "aardvark"
      },
      ...
      "100001" : {
        "animal" : "zebra"
      }
    }
  }
]

And here is an example of the json data stored locally:

[
    {"my_id"=1, "fb_id"=346, "animal"="bear"},
    ...
    {"my_id"=201, "fb_id"=45663, "animal"="water buffalo"}
]

Now I understand that I could simply loop over each item and then make an Angularfire2 "object" call (or even an http.get) in order to see if the item exists in the database:

this.data = [{...},...,{...}]; // local json data
for (let i in this.data) {
  this.fbData.object('animals\' + this.data[i].fb_id).subscribe(dataObj => {
    if (dataObj != null){
      console.log('There is a : ', this.data[i].animal);
    }
  });
}

or

this.data = [{...},...,{...}]; // local json data
for (let i in this.data) {
  let url = 'https://project-1234.firebaseio.com/animals/' + this.data[i].fb_id + '.json';
  this.http.get(url).map(res => res.json()).subscribe((dataObj) => {
    if (dataObj != null){
      console.log('There is a : ', this.data[i].animal);
    }
  })
}

Question:

Although this is the only way I can think of to achieve my goal, even to my intermediate understanding this seems a little labor intensive not to mention slow and getting slower the more records are held locally; so I am wondering if there was a better solution out there or a best practice that can be used in a case such as this?

Many thanks in advance for any help/suggestions.

score 0 · Accepted Answer · edited May 23 '17 at 12:17

An AngularFire2 (or Firebase SDK) implementation will be much more efficient that an HTTP implementation, as the requests will be pipelined -see this answer for more detail.

To retrieve an array of booleans indicating whether or not an item's key exists, you could do something like this:

import 'rxjs/add/operator/concatMap';
import 'rxjs/add/operator/first';
import 'rxjs/add/operator/map';
import 'rxjs/add/operator/toArray';

Observable.from(items)
  .concatMap(item => this.fbData.object(`animals/${item.fb_id}`).first().map(Boolean))
  .toArray()
  .subscribe((result) => console.log(result));

Note that the first operator is used to retrieve a single value from the AngularFire2 object observables. Without the first operator, the object observables would not complete and would continue listening for and emitting changes.

It's possible that this implementation might be acceptable. You say "seems a little labor intensive not to mention slow and getting slower", but have you profiled it?

Without knowing more about your application, it's difficult to offer recommendations. For example, you mention its "getting slower the more records are held locally", but once you've checked whether or not a record exists in the database, can you not store knowledge of that locally? If you are able to do that, perhaps you don't have a problem?

Using AngularFire2 to check if large amount of data exists in Firebase database

Question:

1 Answers1