1

How would one get whole query size efficiently from the Firestore collection which has thousands of documents?

In my case I query documents by few different rules:

  • Start date
  • End date
  • Place id
  • Keywords

Then I limit the query to show only 50 records but I would need to get the size of the query without this limitations since that way pagination would show correctly in the front end.

I could use cloud function which makes the same query as earlier but without limit and then get size of it, but is there more efficient way of doing this? Query size could be thousands of documents so is there any performance issues by doing it this way? And how does the billing work on this kind of situation?

If ie. My query is 1500 documents is there going to be 1500 read operations to get the size of this query?

There has been other topics which recommends using counters to get size of the collection, but this does not suit my approach since the size depends on user's search parameters stated above.

All recommendations for this problem are welcome!

Eljas
  • 1,187
  • 4
  • 17
  • 37

1 Answers1

4

If you have in one collection thousands of documents it might be possible to need to update a counter very often. In Cloud Firestore, you can only update a single document about once per second, which might be too low for some high-traffic applications.

Query size could be thousands of documents so is there any performance issues by doing it this way?

No, it won't. According to the official documentation regarding Firestore counter, you can use distributed counters:

To support more frequent counter updates, create a distributed counter. Each counter is a document with a subcollection of "shards," and the value of the counter is the sum of the value of the shards.

This practice can help you achieve what you want.

And how does the billing work on this kind of situation?

In case you want to read the entire collection at once, you'll be billed with a read operation for each document read.

My query is 1500 documents is there going to be 1500 read operations to get the size of this query?

If you are looping the entire collection to get the number of documents, yes.

For more details about storing counters, please see the last part of my answer from this post:

As a personal hint, don't store this kind of counters in Cloud Firestore, because every time you increase or decrease the counter will cost you a read or a write operation. Host this counter in Firebase Realtime database at no cost.

Alex Mamo
  • 130,605
  • 17
  • 163
  • 193
  • How would you use counters in dynamic searches? For example if user searches every document that has been created in January he would get size of the query and if user includes last December to search he would get different size. And same time user could search documents from 12/12/2018 to today and again he would get different size of documents for this query. I can't see how creating and updating counters would work on this. – Eljas Jan 24 '19 at 13:51
  • Great answer Alex! @Eljas: I highly recommend also checking out the [Getting to know Cloud Firestore](https://www.youtube.com/playlist?list=PLl-K7zZEsYLluG5MCVEzXAQ7ACZBCuZgZ) video series. – Frank van Puffelen Jan 24 '19 at 14:13
  • In this case, you should denormalize the data in Firebase realtime database. This means that if you have a collection of 1500 documents, the same collection should also exist in FRTD. Each child in FRTD should only contain a date, so you can search for objects according to that property. So it's best to count children in FRTD rather than in Firestore where thigs are a little costly when it comes to something like this, right? – Alex Mamo Jan 24 '19 at 14:13
  • 1
    @FrankvanPuffelen Yey! Thanks again puf! :) – Alex Mamo Jan 24 '19 at 14:17
  • @AlexMamo Hey, it helped me to realize that I probably should use both databases together. Now I just were thinking if I should put everything in FRTD and only use that one since there will be "export" button which will export whole query without 50 item limit what I use in frontend to excel. Or should I put same data to both and use Firestore for frontend to load 50 items at a time and then use FRTD to get size of the dataset searched and for exports? Any thoughts? – Eljas Jan 25 '19 at 15:41
  • @Eljas That's the best approach, use them both. If you put all the data only in FRTD, you'll lose all the features that Firestore provides. You can choose to store data in the way is more easy for you but getting data in smaller chunks it's also a very good approach. – Alex Mamo Jan 25 '19 at 15:52