2

I've read all of the questions below, and cannot find anything in the docs to describe how to sync a collection and receive only changed documents from a collection. I've got over 500 documents in my synced collection (using redux-saga-firebase syncCollection) but usually only about 100 of them ever change. At any given time, even fewer will change, but I'm currently getting all 500+ documents back when even one changes, which results in 500+ reads. Not ideal, and will cost me at scale.

Here is my code using redux-saga-firebase, but a vanilla firebase code answer would also suffice. How can get a synchronized response that sends only the changed documents in the collection?

export function* getPlayersFromDb() {
  yield fork(
    rsf.firestore.syncCollection,
    'players',
    { successActionCreator: response => ({
            type: t.GET_PLAYERS_FROM_DB_SUCCESS,
            payload: [...response],
        }),
        failureActionCreator: () => ({
            type: t.GET_PLAYERS_FROM_DB_FAILURE,
            payload: [],
        }),
        transform: response => messageTransformer(response),
    }
);

}

Does Firestore sync only diff?

Realm syncing with large collection on Firestore - architectural questions / issues

How to avoid unnecessary Firestore reads with Cache

What exactly does Firestore synchronization do?

jmargolisvt
  • 5,722
  • 4
  • 29
  • 46

2 Answers2

5

Firestore queries don't have a notion of querying "only things that have changed". You will need to create that notion on your own. The usual way to do this is by recording a timestamp in each document with the time it was last updated, then using that field in a range filter to get only documents updated since the latest update was received. You will need to track that latest update timestamp in the client. One possibility is to use the greatest timestamp seen in any document to date.

Doug Stevenson
  • 297,357
  • 32
  • 422
  • 441
  • Thank you for all your help as always, Doug. Does your suggestion work with synced data or would it only work in a get-where query? I'd like to not have the user need to initiate a request. – jmargolisvt Jun 10 '20 at 23:33
  • I'm not sure what you're asking. It doesn't matter where the query comes from. It just needs to have the correct filter asking for documents whose updated date is greater than the last seen date. – Doug Stevenson Jun 10 '20 at 23:36
  • My current implementation doesn't require sending anything (like a timestamp) from the client. When the db data changes, it just pushes the collection. Are you suggesting I add a range filter to my syncCollection query? Or that this approach won't work and I basically need to set up polling on the client? – jmargolisvt Jun 10 '20 at 23:56
  • Yes, add a filter to the query. You can still listen to results in real time. – Doug Stevenson Jun 11 '20 at 02:05
  • Wouldn't the query need to change frequently to reflect the most recent update value? Is there a way to have the query parameters change, or do you need to re-subscribe? – Nathan Manousos Apr 06 '22 at 18:58
1

This is a great article explaining some tips.

A strategy here is to add a property on each document such as lastModified and .get() from the Firebase client cache and only query the server to get documents with a lastModified greater than the client cache document fetch timestamp.

https://medium.com/firebase-tips-tricks/how-to-drastically-reduce-the-number-of-reads-when-no-documents-are-changed-in-firestore-8760e2f25e9e

Cory Robinson
  • 4,616
  • 4
  • 36
  • 53