0

I am trying to reduce the number of firestore reads by reading from my cache first and then creating a subscription listener based on the lastCacheDate in my cache.

This all works great as long as the app is closed and then reopened, because the subscription is rebuilt. When the app is left open, but not used, the app will read from the server for all new items using the old lastCacheDate the next time it is opened. Is there a different way I should create the subscription or listener or is there a trigger or a way to know the listener is not active? Then I could maybe close the listener and create a new listener with the new lastCacheDate in the cache.

I don't have the option on using StreamBuilder or other widgets, this code is all happening in a provider. Any help would be appreciated.

One thing I haven't tried is to use the WidgetsBindingObserver and figure out how long the app has been inactive, then if it is longer than 30 minutes, close the subscription and reopen with new lastCacheDate when it is resumed. Seems like there should be a better way than that though.

    _initializeData() {
        String CollectionKey = 'collection1';
        String UidKey = 'uidKey';
        String uid = 'uid1';
    
        //query the cache first
        var ref = FirebaseFirestore.instance
            .collection(CollectionKey)
            .where(UidKey, isEqualTo: uid);
        var docSnap = await ref.get(const GetOptions(source: Source.cache));
        for (final document in docSnap.docs) {
            _fireLog.info('Retrieved cache document');
            _processDoc(document.data());
        }
    
        //create a subscription to listen for new items
        _allItemsSubscription = FirebaseFirestore.instance
            .collection(CollectionKey)
            .where(UidKey, isEqualTo: uid)
            .where(ItemInterface.ModifiedDateKey,
                    isGreaterThan: LocalRepositoryManager.itemCacheDate)
            .snapshots()
            .listen((snapshot) {
                    for (final document in snapshot.docs) {
                    _fireLog.info('Retrieved server document');
    
                    // this processDoc will set the LocalRepositoryManager.itemCacheDate
                    _processDoc(document.data());
                    }
                    }
            );
    }
Brian Elkin
  • 50
  • 4
  • 1
    "When the app is left open, but not used, the app will read from the server for all new items using the old lastCacheDate the next time it is opened." I'm not sure I understand the problem here. Isn't that exactly what you want? What type of updates are you missing, or where are you experiencing unexpected reads with this approach? – Frank van Puffelen Mar 21 '23 at 23:42
  • @FrankvanPuffelen, if I have 100 items on the server and 1 item in the cache, when the app is started, I will read all 100 items and put them in the cache and update my lastCacheDate. The issue is that when the client is idle for more than 30 minutes, when the client is open again, the server will again read all 100 items because the subscription / listener is using the old lastCacheDate. How do I know when to cancel the old subscription and create a new one with the new lastCacheDate? – Brian Elkin Mar 22 '23 at 02:07
  • How about updating that value right after you get a new document, and then canceling the existing listener and starting a new one? – Frank van Puffelen Mar 22 '23 at 13:34

2 Answers2

0

I think firestore plugin already take care of what you want to achieve, See Questions about firestore cache and reads charge

flutroid
  • 1,166
  • 8
  • 24
  • From my understanding after 30 minutes the firestore plugin will re-read all items in the subscription / listener. I don't know the best way to update my subscription with the new lastCacheDate. Do I need to cancel it and then create it again if the client has been idle for longer than 30 minutes? – Brian Elkin Mar 22 '23 at 02:09
  • If you mean when after cancel listener 30 mins then reattach again, see [Is Firestore charging us when offline more than 30 minutes?](https://stackoverflow.com/questions/53002397/is-firestore-charging-us-when-offline-more-than-30-minutes) – flutroid Mar 22 '23 at 02:20
0

I'd probably:

  1. Update the LocalRepositoryManager.itemCacheDate value every time you get an updated snapshot from the server.
  2. Then cancel the existing listener.
  3. And start a new one.

That way you're always listening for data that the client doesn't have in its cache yet. That should minimize the number of document read charges.

Frank van Puffelen
  • 565,676
  • 79
  • 828
  • 807
  • So I could do that, but closing and opening listeners constantly is not something they really recommend. I will start to look into using the WidgetsBindingObserver to see if I can close and reopen when the app resumes after a certain amount of time. – Brian Elkin Mar 22 '23 at 17:41
  • "they" -> you might want to check my profile . --- If you continuously get a lot of updates (instead of bursts), you can also limit the number of times you reattach by [debouncing](https://www.techtarget.com/whatis/definition/debouncing) it. That will then lead to potential doubly reading some docs again though, so that's a trade-off you'll have to make. – Frank van Puffelen Mar 22 '23 at 22:38