Firebase: how to retrieve only changed items next time app launches?

Question

I've got an app that uses a Firebase db containing 100,000 items. My app has to process through each of these items which takes several seconds. What is happening is that every time the app is launched (from a terminated state) those 100,000 items are being processed each time (even if the contents of the db on the Firebase server have not changed). Obviously, I don't want the app to do this if not necessary. Here's some code:

if dbRef == nil {
    FirebaseApp.configure();
    Database.database().isPersistenceEnabled = true
    ...
    let dbRef = Database.database().reference(withPath: kFirebaseDBName)
    _ = spamRef.observe(DataEventType.value, with: { (theSnapshot) in
        if let content = theSnapshot.value as? [String : AnyObject]
        {
            self.processContent(content: content)
        }

Each time the app is started then the content snapshot contains the entire database reference contents.

Is there a way of, for example, getting the last date the database was updated (on the server), or only obtaining the delta of changed items between each app launch - can a query return just changed since last queried for example, or something similar?

I don't know how many items have changed so cannot call something like: queryLimited(toLast: N)) As I don't know what value N is.

I've tried adding keepSynced as follows in the hope it might change things, but no.

if dbRef == nil {
    FirebaseApp.configure();
    Database.database().isPersistenceEnabled = true
    ...
    let dbRef = Database.database().reference(withPath: kFirebaseDBName)
    dbRef.keepSynced(true) 
    _ = dbRef.observe(DataEventType.value, with: { (theSnapshot) in
        if let content = theSnapshot.value as? [String : AnyObject]
        {
            self.processContent(content: content)
        }

I have no idea how much data might have changed so don't know what value to supply to something like toLast or similar to modify the observation parameters. The database (which was not created nor updated with new content by me) has 100,000 items in a flat structure (i.e. one parent with 100,000 children) and any number of these children in any order might have been deleted and replaced since last time my app ran, but the total will still be 100,000. None of the children have an explicit timestamp or anything like that.

I was under the impression if Firebase kept a local cache of the data (due to isPersistenceEnabled) then next time it connects with the server it would only sync what had changed on the server. Therefore in order to do this Firebase itself must internally have some delta information somewhere, so I was hoping that delta information may available in some form to my app.

Note: My app does not need persistence to be enabled, the above code is doing so just as variations to see if anything will result in the behavior I desire with the observer.

Answer 1

UPDATE

So looking at the documentation more you can set a timestamp for the last time a user was connected to the server using:

lastOnlineRef.onDisconnectSetValue(ServerValue.timestamp())

Take a look at this question Frank explains some issues with persistence and listeners. The question is for Android but the principles are the same.

I still think the problem is your query. Since you already have the data persisted .value is not what you want since this returns all of the data.

I think you want to attach a .childChanged listener to your query. In this case the query will only return the data that has been changed. If you haven't heard of .childChanged before you can read about it here.

I didn't realize this problem is specifically related to persistence. I think you are looking for keepSynced(). Take a look at this.

ORIGINAL ANSWER

The problem is your query. You are asking for all of the data that's why you're getting all of the data. You want to look into limiting your queries using toFirst or toLast. Additionally, I don't think you can query for the last time the database was updated. You could check the last node in your data structure if you have the timestamp saved, but you might as well just get the newest data.

You want something like this:

ref.child("yourChild").queryLimited(toLast: 7).observeSingleEvent(of: .value, with: { snap in
    // do something
})

Depending on how you're writing your data you'll want toLast or toFirst. Assuming the newest data is written last toLast is what you want. Also note that the numbers I am limiting to are arbitrary you can use any number that fits your project.

If you already have a key and you want to start querying above that key you can do something like this:

ref.child("YourChild").queryOrderedByKey().queryEnding(atValue: lastVisiblePostKey).queryLimited(toLast: 8).observeSingleEvent(of: .value, with: { snap in
    // do something with more posts
})

You may also want to look into this question, this question and pagination.