Queries depending on dataset in Firestore

Question

Recently I have migrated from firebase realtime database to firebase firestore because of the fact that it says the speed of the query depends on the size of the dataset(number of documents) being retreived from collection and not on number of documents in a collection. I checked with varying number of documents in a collection 100, 5000 and 10000 and I was retreiving 20 documents in one query. What I saw was the resulting time for the query increased when I moved from 100, 5000 and 10000 documents in a collection. Why is this happening ? Is it because firestore is in beta ?

Querying on android (collection with 21000 docs)

   collectionReference= FirebaseFirestore.getInstance().collection("_countries").document("+91").collection("_classes-exams").document(String.valueOf(mItem)).collection("_profiles").document(mOtherUserUid==null?uid:mOtherUserUid).collection("user_data");


collectionReference.document("_content").collection(articlesOrQuestions)

                    .orderBy("mTimeStampAsDate", Query.Direction.DESCENDING).limit(20).get().addOnCompleteListener(mCompleteListener)

                    .addOnFailureListener(new OnFailureListener() {

                        @Override

                        public void onFailure(@NonNull Exception e) {

                            Toast.makeText(getContext(), e.getMessage(), Toast.LENGTH_LONG).show();

                        }

                    });

Image of android monitor when querying the above collection reference: https://i.stack.imgur.com/QZaVX.jpg

You can see the query took almost one minute by looking at the heap(after one minute memory didn't changed much and remained constant and there is sudden spike in network section after 1 minute by which you can infer onComplete is called). What is happening between calling 'get()' function and 'onComplete' callback. This doesn't happen when querying small collections. and why the query on web is fast but slow on android ? Link to jsbin: http://jsbin.com/fusoxoviwa/1/edit?html,css,js,console,output

Most likely you're seeing the time it takes to **transfer** the data to your client, which is limited by the bandwidth of your device. But without seeing the minimal code that reproduces the behavior, it's hard to say anything conclusive. — Frank van Puffelen, Feb 05 '18 at 15:04
But I am receiving 20 documents in all three cases and documents are similar so the data being received by the client is same in all three cases. Is there some additional data other than documents(20) received when the number of documents in a collection is large. Should I show the way I am querying ? But how can I show you the dataset is different ? @FrankvanPuffelen — user9290695, Feb 05 '18 at 15:11
Oh right, same number of documents. Sorry, it's still a bit early here. :-) I haven't tested the "performance depends only on size of result set" much yet. Can you set up something like a jsbin that reproduces the difference, so I can have a look? — Frank van Puffelen, Feb 05 '18 at 15:37
I have developed my app only for android till now and I don't know how to set up jsbin with my firebase project. We can have a teamviewer session so that you can have a look and check the difference. If you can only check with jsbin, it can take some time for me to be familiar with it. @FrankvanPuffelen — user9290695, Feb 05 '18 at 15:46
I think sharing the html and javascript in jsbin will also include the api key which I don't know is OK to share. Although I tried querying on web and the queries ran fine and all the queries took almost same time but running the same query on android is causing havoc. There is so much memory being allocated on heap when querying a collection having 10000 documents for 20 documents. Around 50 mb of heap is allocated. Even I checked in the logs that only 20 documents are present in QuerySnapshot and the time taken to query is one minute. I checked this by the difference between onComplete callb — user9290695, Feb 05 '18 at 17:19
Continued- ack and the time just before calling get() and adding completion listener statement. Why is this happening on android ? @FrankvanPuffelen — user9290695, Feb 05 '18 at 17:21
Sharing an API key is not a security concern. See https://stackoverflow.com/questions/37482366/is-it-safe-to-expose-firebase-apikey-to-the-public. If you can't set up a jsbin, at least share the code that you used for testing. You questions are very concrete, but I need to see the code that you're asking about to be able to answer. — Frank van Puffelen, Feb 05 '18 at 18:02
@user9290695 is the account I logged in from mobile which I can't login from web now. The link to jsbin is: http://jsbin.com/hodezuquwi/2/edit?html,css,js,console,output. You can check that the time is almost same on web. In some time I will be adding the android code because that is causing problem. — Kartik Watwani, Feb 05 '18 at 18:30
Sorry, I had missed your previous updates. Do you mean that the problem cannot be reproduces with the Web SDK? In that case the only thing I can think about is that the time is going into decoding the JSON, but 1m for 20 docs sounds quite excessive there. What does `mCompleteListener` look like? — Frank van Puffelen, Feb 06 '18 at 04:33
I think yes I tried using web sdk and there was no problem at all. Which JSON are you talking about ? The one which is received from firestore? If yes, I don't think so because In the android monitor(network section) there is spike after one minute which I think is the time data is received from firestore and after that it doesn't take much time to put data into recycler view. 'mCompletionListener' is simple updating the underlying arraylist for recyclerview and updating the cursors for paginating and minor UI updates like changing progressbar visibility @FrankvanPuffelen — user9290695, Feb 06 '18 at 04:50
And the time spent in Android (the 1m you mention) goes up as you query a larger collection? — Frank van Puffelen, Feb 06 '18 at 19:29
I haven't checked with a collection greater than 21345 documents. I have checked it for collections lesser than the above number and yeah as the number of documents starting from 100 to 5000 to 21000 the time did increase. In the collection which has 100 documents the time spent is not more than 5 seconds. @FrankvanPuffelen — user9290695, Feb 06 '18 at 19:36
OK. Thanks for sticking with me here. I've asked some team mates to have a look, to see if they can make sense of it. — Frank van Puffelen, Feb 06 '18 at 19:38
Did you write these collections from the same Android client that's now loading a subset of documents from them? If so, that would explain. The client-side cache in that case will contain information about all docs, and your app is spending time churning through that information. If you try on a "clean client", it won't have any information cached and it should only spend time on documents that the client is requesting (or has requested before). — Frank van Puffelen, Feb 06 '18 at 20:02
Yes I did wrote all these collections from the client I am querying the subset on. I cleared the app data from settings and voila it loaded very quickly. Thank you so much @FrankvanPuffelen — user9290695, Feb 06 '18 at 20:07
Any updates on the Garbage Collection, or the faster indexes from firestore? — Jeff Padgett, May 01 '18 at 20:44

score 2 · Accepted Answer · answered Feb 06 '18 at 21:20

Did you write these collections from the same Android client that's now loading a subset of documents from them? If so, that would explain.

The client-side cache in that case will contain information about all docs, and your app is spending time churning through that information.

If you try on a "clean client", it won't have any information cached and it should only spend time on documents that the client is requesting (or has requested before).

The behavior you're currently seeing should improve before Firestore exits beta, since the index will become more efficient, but also because it'll get some form of GC.

Queries depending on dataset in Firestore

1 Answers1