Sum firebase data efficiently up

Question

I have an app where I have up to a few thousand entries in a firebase table. Now someone is connecting to this table and I need to count and sum these items up.

-itemlist
|
--1
  |-x:2
--2
  |-x:4

and so on. A user can connect at any time (meaning he never saw this table before) and another user can change some value at any given time or add values.

Now if I try and add a ValueEventListener to "itemlist" it works if there is no new data but as soon as I connect to a new database with 1000 entries or so the listener calls for every one of these 1000 entries.

I tried with onChildlistener but this calls the on ChildItem added for every item extra on the first time. This takes over 2 minutes sometimes.

Is there a way to get the whole database once, calculate it and then only listen for changes?

Answer 1

The Firebase Database has no server-side aggregation operators, since they would inherently be at odds with the scalability and realtime nature of the database.

That leaves you with two options to calculate aggregates:

1. perform them client side as you already suggested

2. keep a running aggregate that you update with every change

Performing the aggregation client-side is a good option if the client already needs to display the data anyway. For example: if you're showing the list of users in a specific chat room, you can easily count the number of users client-side from the same data.

But if you don't need the data client-side, then just downloading it to aggregate it is wasteful and will hurt scalability of your app. In such cases, an option is to keep an extra node that keeps the aggregate and update it with every relevant write operation. For example: if you want to show how many users have registered with your app, you could keep a global /user_count that you update whenever a user registers/unregisters. For this update, you'd typically use a transaction.