How to reduce the number of reads in a Firestore collection?

Question

I've learned how to use pagination from a previous question of mine and I've implemented it in my a dictionary app for maritime terms. There are plenty of terms in the database.

The db is like so:

maritime_terms
  |- id
      |- term: "broad reach"
      |- details: "Sailing with the wind abaft the beam."
      |- etc

As in the docs I use:

db.collection("maritime_terms")
        .startAt(searchText)
        .endAt(searchText+ "\uf8ff")
        .limit(20);

When I start to search for example "broad reach" and I type the first letter b, I get 20 results. When I type the second letter, I get other 20 results. And so on. The problem is that for each letter I type I get 20 reads, so for a simple search I get from minimum of 100 reads to 150 or even more.

So far my app works great but I think it's expensive for a simple dictionary app. How can I reduce this huge number of reads?

Answer 1

While Fogmeister's answer can help you reduce the number of reads significantly, I'll try to provide you another approach, but it will imply a change in your database structure. So this is what I would do.

First of all I will stop searching the maritime_terms collection in the way you do. Instead of that, I will create a document that will hold all the terms that you have in the database. Every term will be added in an array of terms. According to the length of the terms, it's possible to fit in one document all terms. I say that because the documents have limits. So there are some limits when it comes to how much data you can put into a document. According to the official documentation regarding usage and limits:

Maximum size for a document: 1 MiB (1,048,576 bytes)

As you can see, you are limited to 1 MiB total of data in a single document. When we are talking about storing text, you can store pretty much.

If all the terms do not fit in a single document, then you can simply create a document for each letter from the alphabet. Since you know what you search, it will be very easy to know what document to read. So take a look at the following schema:

Firestore-root
   |
   --- alphabetTerms (collection)
          |
          --- a (document)
          |   |
          |   --- aTerms: ["A-Term", "A-Term", "A-Term"]
          |
          --- b (document)
              |
              --- bTerms: ["B-Term", "B-Term", "B-Term"]

To get the a document, simply get a reference and make a get() call:

FirebaseFirestore rootRef = FirebaseFirestore.getInstance();
DocumentReference aRef = rootRef.collection("alphabetTerms").document("a");
aRef.get().addOnCompleteListener(/* ... */);

Even if the the aTerms is an array, when you make a get() call, you'll get it as a List<String> and not as an array. Then you can simply search that list of strings using:

termFromTheList.contains(searchedTerm);

One thing to note is that all terms must be unique in order to use this solution.

This solution it's a much better solution than the one you are using right now since it allow you to search a substring. For instance, if you want to search in broad reach for reach, this is not possible, unless you are using a third party service like Algolia.

Once you find a match, get it and create a new query:

FirebaseFirestore rootRef = FirebaseFirestore.getInstance();
CollectionReference maritimeTermsRef = rootRef.collection("maritime_terms");
Query searchQuery = maritimeTermsRef.whereEqualTo("term", searchedTerm);
searchQuery.get().addOnCompleteListener(/* ... */);

Using the solution above, you'll ending up having only one document read for finding the term in the document and one document read for displaying the details of the term.

So you'll reduce the number of reads from 150 to only 2.

Answer 2

The way to do this is to debounce the search.

This means that you can exploit the fact that a user will be typing at a certain number of key presses per second and delay the search a short while after each key press.

So for example if the users can be expected to type at a rate of 5 key strokes per second then after each key stroke start a timer of, say 0.3 seconds, only after this timer has expired will it start the search.

If the user taps another key in less than 0.3 seconds then scrap the timer and start it again for another 0.3 seconds.

This way, only once the user has stopped typing (or at least stoped typing for now) will the search actually be executed with the whole string.

However, this is not a change in Firestore. This would be a change in your client logic.

You can read more about debouncing (and also throttling) at this link.

Example:

1. User types B: start timer for 0.3 seconds.
2. User types r 0.2 seconds later: cancel timer, start a new timer for 0.3 seconds.
3. User types o 0.1 seconds later: cancel timer, start a new timer for 0.3 seconds.
4. 0.3 seconds elapsed with no typing. Start search with text Bro.