Firestore Query - OrderBy two fields / order by radius AND timestamp

Question

Data structure:

To stick with Todd Kerpelman's examples:

Let's say I'm trying to get all restaurants within a certain radius ordered by a timestamp field (e.g. the time they last updated a menu).

I also need pagination for that query, so I need 3 restaurants at first, then 3 more etc. ordered by their timestamp field.

I found GeoFire for Android which uses a GeoHash algorithm to store GeoPoints (coordinates) as numbers in a field called geoFireLocation. Now I can query the closest restaurants within a certain radius by calling:

.whereNearTo(queryLocation, searchDistance)

which under the hud is implemented like this:

public GeoFireQuery whereNearTo(QueryLocation queryLocation, Distance distance) {
    BoundingBoxUtils geoPointUtils = new BoundingBoxUtils(distance.getUnit());
    BoundingBox boundingBox = geoPointUtils.getBoundingBox(queryLocation, distance.getDistance());
    this.query = this.query
        .orderBy("geoFireLocation")
        .whereGreaterThanOrEqualTo("geoFireLocation", boundingBox.getMinimumMatch())
        .whereLessThanOrEqualTo("geoFireLocation", boundingBox.getMaximumMatch());
    return this;
  }

This is working fine so far.

Now I'm trying to order by that timestamp field like this:

.orderBy("timestamp").whereNearTo(queryLocation, searchDistance)

Firestore tells me:

You have an inequality where filter (whereLessThan(), whereGreaterThan(), etc.) on field 'geoFireLocation' and so you must also have 'geoFireLocation' as your first orderBy() field , but your first orderBy() is currently on field 'timestamp' instead

Trying to order by the timestamp after the geoFireLocation seems to be totally ignored by firebase:

.whereNearTo(queryLocation, searchDistance).orderBy("timestamp")

Printing the timestamps of the documents in the order I receive them:

        for (DocumentSnapshot documentSnapshot :
                a.getDocuments()) {
            Log.e("MY_LOG", documentSnapshot.toObject(EventPOJO.class).getTimestamp().toString());

In the console I see that the documents are not ordered by timestamp:

E/MY_LOG: 2020-12-17T23:00:00.000+01:00
E/MY_LOG: 2020-12-13T23:00:00.000+01:00
E/MY_LOG: 2020-12-15T23:00:00.000+01:00
E/MY_LOG: 2020-12-14T23:00:00.000+01:00
E/MY_LOG: 2020-12-09T23:00:00.000+01:00

I know I can still sort my results by their timestamps on the client-side. This can lead to weird flipping effects when using an endless scrolling recycler view.

I could get the 3 closest restaurants and sort them by their timestamp, then get another 3 and merge them with the last 3 and sort again. Worst case all the previous restaurants jump to the bottom of my list if their timestamps are all older than the ones from the new restaurants received.

Is there anything I can do in this situation?