9

Suppose I'm using firebase for a commenting system and I want to retrieve the comments for a given topic, but there are so many comments in one topic that I don't want to retrieve them all at once. I also want the newest comments to be displayed on top.

It seems that the only way to display firebase records in reverse order is to retrieve them all and then iterate over them in reverse.

This could get very unwieldy on large datasets, especially for mobile clients.

Is there any better way? What is the generic and preferred solution to query paginated data from Firebase, in ascending or descending order?

Pandaiolo
  • 11,165
  • 5
  • 38
  • 70
cayblood
  • 1,838
  • 1
  • 24
  • 44
  • "It seems that the only way to display Firebase records in reverse order is to retrieve them all and then iterate over them in reverse." You might want to re-read the documenation, specifically about `limit`. https://www.firebase.com/docs/web/guide/retrieving-data.html#section-queries – Frank van Puffelen Nov 28 '14 at 23:35
  • Thanks Frank, I was aware of those docs. Priority can be used to sort in different orders, but I'm having trouble thinking of an effective way of setting an item's priority in a way that would cause it to be sorted in reverse chronological order. Can you? – cayblood Nov 30 '14 at 05:19
  • 1
    Not just by priority, you can nowadays sort by any child using `orderByChild`. Read this blog post to catch up: https://www.firebase.com/blog/2014-11-04-firebase-realtime-queries.html. Something like `orderByChild` followed by `limitToLast` and then `Array.reverse` should do what is needed for you. – Frank van Puffelen Nov 30 '14 at 13:36
  • Thanks, limitToLast was the missing piece of the puzzle for me. I could see how to get all the comments and then reverse them but this allows things to scale up when there are many comments. – cayblood Dec 01 '14 at 18:19
  • OK. Feel like self-answering the question? Preferably with some code that shows how you solved the problem. – Frank van Puffelen Dec 02 '14 at 02:15
  • Thanks Frank. I'm scouting out this problem in advance. I haven't implemented it yet, but your comments will at least help me do so. When I get to it, I'll post my answer. My current naive implementation retrieves all comments, but it doesn't scale. – cayblood Dec 02 '14 at 17:52
  • 1
    See also: [Paginate](http://firebase.github.io/firebase-util/#/toolbox/Paginate/example/); in [firebase-util](http://firebase.github.io/firebase-util/#/). – Kato Nov 03 '15 at 18:39

2 Answers2

14

Updated answer

@Tyler answer stackoverflow.com/a/38024909/681290 is a more up to date and more convenient answer for this question

TL;DR

There is no solution if what you are looking to do is :

ref.orderBy(field, 'DESC').offset(n).limit(x)

Also, in Firebase github there are unsupported tools that do pagination, although only in ascending order only.

Otherwise, here are the closest possible solutions, that I have found myself, or on the web so far. I deliberately interpreted the question as generic, not only about time fields as asked by the OP.


Use priorities

The goal is to use setWithPriority() and setPriority() on children to be able to get ordered data later with orderByPriority().

Issues :

  • I don't see the advantage over using an indexed priority field instead ? (in fact, priority is stored as an underlying field called .priority, that you can see when exporting json data)
  • Hard to maintain for many use cases, to determine value of priority to set

Redundant child fields with negative data

For example, we can use a negative timestamp timeRev indexed field in addition to time to be able to get the newest items first.

ref.orderByChild('timeRev')
   .limitToFirst(100);

Issues :

  • It adds more complexity to the app : additional fields need to be maintained
  • Can break atomicity of the field (for example a score field can be updated at once, not sure if it's possible with two fields, one positive and one negative)
  • I think this workaround was used when there was only limit() in the Firebase API, but is kinda obsolete now that we can use limitToFist(), limitToLast(), and range queries (see below)

Using limitToLast() and endAt() range queries

This let us avoid negative and redundant field :

ref.orderBy('time')
   .limitToLast(100)

This should be pretty effective with timestamp fields, because usually this is a unique field.

The resulting array of items simply need to be reversed before use. (just remember Array.prototype.reverse() is mutable, so it will change your array)

Issues :

  • The API docs say that only the ordered key value can be set as a startAt or endAt boundary. If a lot of items share the same value, the dataset cannot be split in fixed length offsets.

Example with the following items holding a score value :

{
  items: {
    a: {score: 0},
    b: {score: 0},
    c: {score: 1},
    d: {score: 1},
    e: {score: 2},
    f: {score: 5}
  }
}

First page query to get the best scoring items :

ref.child('items').orderByChild('score')
   .limitToLast(3)

Results :

{
  d: {score: 1},
  e: {score: 2},
  f: {score: 5}
}

Note the first item of the subset has a 1 score, so we try to get the previous page by selecting all items with score of 1 or less :

ref.child('items').orderByChild('score')
   .endAt(1)
   .limitToLast(3)

With that query, we get b,c,d items, instead of a,b,c items, which is expected as per the API docs, given the endAt(1) is inclusive, so it will try to get all scores of 1, and has no way to sort which were already returned before.

Workaround

This can be mitigated by not expecting each subset to hold the same amount of record, and discarding those which have already been loaded.

But, if the first million users of my app have a 0 score, this subset cannot be paginated, because the endAt offset is useless, as it is based on the value instead of the number of records.

I don't see any workaround for this use case, I guess Firebase is not intended for this :-)

Edit: In the end, I am using Algolia for all search-related purpose. It's gt a really nice API, and I hope google ends up acquiring Algolia to integrate both Firebase & Algolia, as their complementarity is close to 100%! Disclaimer: no shares owned!! :-)

Community
  • 1
  • 1
Pandaiolo
  • 11,165
  • 5
  • 38
  • 70
  • 1
    This is driving me crazy, in reversed order i can't get elements for my page 2. Say I have 100 elements per page, it's easy to get the first page with `limitToLast` 100 and reverse array. Now how do I get second page? My keys are firebase generated keys `-Jp9p7glAp98TMYBOcig` etc. Only if Firebase would give me total number of elements in a node without pulling everything from it, that would be a breeze. I have 117k elements in a node and it weights 25mb (with shallow 3mb), so it's not a solution. – Skyzer Dec 15 '15 at 00:37
  • I wish there was `startAt`/`endAt` in combination with limit to retrive specific number of elements. So it could be done with first page using `limitToLast` 100+1 elements and then using the 101st element key to get elements 101-200 with `endAt=101st element` and limiting result for 100 elements. – Skyzer Dec 15 '15 at 01:29
  • `starAt` content *must* match the `orderBy`. for example in reverse order : `.orderByKey().endAt(lastKnownKey).limitToLast(100)`. *Edit: looks like it is what you are doing... well it works for me :)* – Pandaiolo Dec 15 '15 at 11:37
  • Seems like it's doable, just double checked and returns indeed elements from 100-200. But because I'm using REST API, can't see any elegant way to save the `lastKnownKey` for the next request. It's not a form submission, so no hidden element, it's just next/previous links, I guess have to go with passing `lastKnownKey` to into query string. – Skyzer Dec 15 '15 at 12:29
  • I am trying to apply the solution using iOS API. Calling queryEndingAtValue, queryLimitedToLast you will get a returning result type of NSDictionary not NSArray. How can we reverse the elements reliably at this stage? Any suggestions? – Peter Peng Nov 30 '16 at 21:10
  • Never mind, I found a solution. You can get a NSArray result from "snapshot.children.allObjects" – Peter Peng Nov 30 '16 at 21:22
-1

Here is solution from firebase docs:

Query query = cities.orderBy("name", Direction.DESCENDING).limit(3);

https://firebase.google.com/docs/firestore/query-data/order-limit-data

Rakesh
  • 522
  • 4
  • 13