Realm syncing with large collection on Firestore - architectural questions / issues

Question

I have a collection of product objects (title, desc, price, quant, urlString, etc) in a Firestore collection. Currently around 1000 items, but that could go to 10k. On my iOS app launch, I setup a collection listener (db.collection("products").rx.listen()) which then saves changes to a local Realm database.

.subscribe(onNext: { querySnapshot in
     querySnapshot.documentChanges.forEach { docChange in
                autoreleasepool {
                    let realm = try! Realm(configuration: Realm.Configuration.defaultConfiguration)
                    let newData = docChange.document.data()
                    if let item = itemFactory.createItem(using: newData) {
                        if (docChange.type == .added) {
                            //realm.add(item)
                        }
                        if (docChange.type == .modified) {
                            //realm.update(item)
                        }
                        if (docChange.type == .removed) {
                            //realm.delete(item)
                        }
                    }
                }
            }
        }, onError: { error in
            print("Error fetching snapshots: \(error)")
        }).disposed(by: disposeBag)

I've read the firestore docs in detail but I'm not 100% confident this approach is reliable or performant.

Question: When the app launches, will Firestore download all 10k documents each time, before describing the changes? Or will it cache all 10k the very first time then only download changes on subsequent launches. I'm confident once a change event has fired, all subsequent events will only pick up changes to the Firestore database. What I'm concerned about is on first subscribing to the listener when the app launches, it downloads all 10k items (which would be costly).

EDIT 9 Jan 2019:

I ended up implementing @zavtra elegant answer with code roughly looking like this:

var newestUpdatedAt = UserDefaults.standard.double(forKey: kUDItemUpdatedAt)
//...
db.collection(kProducts)
            .whereField(kUpdatedAt, isGreaterThan: newestUpdatedAt)
            .rx.listen()
//...

querySnapshot.documentChanges.forEach { docChange in
            autoreleasepool {
                let realm = try! Realm(configuration: Realm.Configuration.defaultConfiguration)
                let newData = docChange.document.data()
                if let item = itemFactory.createItem(using: newData) {
                    if item.updatedAt > newestUpdatedAt {
                       newestUpdatedAt = item.updatedAt
                    }
                    if (docChange.type == .added) {
                        //realm.add(item)
                    }
                    if (docChange.type == .modified) {
                        //realm.update(item)
                    }
                    if (docChange.type == .removed) {
                        //realm.delete(item)
                    }
                }
            }
        }
        UserDefaults.standard.set(newestUpdatedAt, forKey: kUDItemUpdatedAt)

Answer 1

According to the docs:

docChanges returns an array of the document changes since the last snapshot. If this is the first snapshot, all documents will be in the list as "added" changes.

Every time you re-start the app will trigger this "first snapshot" behavior. If you want to get around this behavior, you would have to:

1. Retrieve the most recent document saved locally, with its timestamp.
2. Build a query where all documents start at that timestamp (i.e., every document's timestamp is at a minimum, the most recently saved timestamp)
3. Subscribe to changes on that query on app entry.

To do this, you will have to add a timestamp field to each document, and an "indexOn" rule in your firestore rules on the timestamp field in order to prevent client-side downloading and sorting of the entire collection.

Answer 2

When the app launches, will Firestore download all 10k documents each time, before describing the changes?

When you are listening for changes in Cloud Firestore for realtime changes, using Firestore Query's addSnapshotListener() method, it:

Starts listening to this query.

Which basically means that first time you attach the listener, you get all documents that correspond to that particular query.

Or will it cache all 10k the very first time then only download changes on subsequent launches.

Because Firestore has offline persistence enabled by default it means that once you perform a query, the results are chached on user's device. Furthermore, everytime a property within a document changes, you are notified according to that change. Obviously, this is happening only if the listener remains active and is not removed. So if nothing in the database is changed, you get the entire data from the cache.

As also @zavtra mentioned in his answer, you can add add under each object from your collection a Date property (this is how you can add it) and query your database on client, according to this new property, for all documents that have changed since a previous time.

I also recommend see Doug Stevenson's answer from this post, for a better understanding.