What is the correct with to do pagination with Firebase firestore's querySnapshot?

Question

I am currently dealing with pagination with addSnapshotListener in Firebase's firestore and it appears there's no easy way to implement Snapshot with pagination.

Original premise: I started the implementation with addSnapshotListener as follows:

db.collectionGroup("images")
            .order(by: "createdTime")
            .whereField("featured", isEqualTo: true)
            .addSnapshotListener { querySnapshot, _ in
                if let querySnapshot = querySnapshot {
                    vm.featuredImages = querySnapshot.documents.compactMap { document in
                        do {
                            let image = try document.data(as: ImageModel.self)
                            print("DEBUG FEATURED IMAGE: \(String(describing: image))")
                            return image
                        } catch {
                            print("DEBUG FEATURED IMAGE ERROR \(String(describing: error))")
                        }
                        return nil
                    }
                }
            }

And all goes well. The data is fetched into the ViewModel and any new changes are automatically notified via Firestore’s library and the local model gets updated.

Now add pagination: I’ve read the official documentation as well as all the stackoverflow threads. It appears there is no easy to maintain a addSnapshotListener with a new page.

(A) One naive approach when a new page is requested would be to

1. Keep track of the old listener, and then unregisters the old one
2. Register a new snapshotListener now with a new page (10 -> 20 elements)
3. Repopulate the local model

This seems to work ok on the surface however with the one big problem is that you would be re-fetching the first 10 when you request for page 2. And the fetches become exponential as you add pages!

(B) Another solution mentioned in Firebase’s official youtube is

1. Keep the old listener, but keep adding a new listener per new page
2. On first fetch, it’s easy, you would just dump the new data into the old local model
3. But when things update, it’s a lot of manual work. You would have to either diff the new data vs the old local model or somehow find a way to coordinate all the listeners and merging them into a new modal .

I imagine querySnapshot is the standard way of keep data in sync with apps. I imagine every app is going to need pagination. What is the correct solution?

Answer 1

A. First part is to handle new documents and display document for the first time during the pagination:

1. Use a snapshot listener to pull the latest documents, say the last 10
2. When you detect that the user has seen all the docs you have already loaded, load the next set of documents (say next 20) with a new get() query, separate from the listener

The difficult part here is to handle the case when your app goes offline

1. When back online you have to detect whether more documents have been added than your listener is covering
2. If not more then just append these to the start
3. If more then you have to perform a new query to get the missing ones (and paginate it if say that's 1000 docs). So you can end up with many "holes" in your local list of documents for when you app was offline

B. Secondly, you have to handle updates on documents that are not part of the snapshot listener (all the one after the first 10).

1. In my app, I am making a get query to the backend every time a new such document is scrolled to, checking if it has been upddated since the last time it was loaded. I have optimized this get query as follows:

• whenever a document is updated (including deletions) a field time_updated is set

• when a document is fetched my app stores locally the document's time_updated as last_pulled_updated

• when a user scrolls back to a document my app will fetch it as well as the next 9 with the following query:

  .order(by: "time_updated", descending: true)
  .whereField("time_updated", isGreaterThan: min_last_pulled_updated) // Minimum of last_pulled_updated over the 10 docs
  .whereField("featured", isEqualTo: true)
  .whereField("docid", in: [docid1,docid2,docid3,docid4,docid5,docid6,docid7,docid8,docid9,docid10])
  

• this costs at most 10 reads and may cost only 1 read if one or no document is returned

2. If the user is viewing a given document for some time, I am creating a snapshot listener for it (and the last 4 + next 5 like in 1 for optimisation purpose) to be able to update likes and comments on it in live (it is a social media app)

C. Deletions. By deletion I mean rendered innaccessible to the end user, not deleted from Firestore. So a "deleted" document could simply have a field deleted set to true and the listener and pagination query will both contain.whereField("deleted", isEqualTo: false). You can use TTL to have it really deleted later on.

1. Beyond the first 10, this is taken care of by B1.
2. Within the listener you can deduce when a doc has been deleted: 4th doc disappear, can only be because it was deleted. For the 10th doc there is a special case when a new doc is created and the 10th doc is deleted at the same time - then you cannot know from the listener only if it was deleted or if it was simply pushed out of the listener. For that purpose, in my app, I do an additional query like in B1 every 10 new documents for the documents 11th to 20th (this case is very rare in my app)

Answer 2

I always done this with solution (A) you mentioned.

This seems to work ok on the surface however with the one big problem is that you would be re-fetching the first 10 when you request for page 2. And the fetches become exponential as you add pages!

This only happend when you using .get, But if you are attach listener to it you only get charged from those documents that haven't fetched yet. See this question,