29

I'm testing firebase for a project that may have a reasonably large numbers of keys, potentially millions.

I've tested loading a few 10k of records using node, and the load performance appears good. However the "FORGE" Web UI becomes unusably slow and renders every single record if I expand my root node.

Is Firebase not designed for this volume of data, or am I doing something wrong?

Jay Stevens
  • 5,863
  • 9
  • 44
  • 67
winjer
  • 301
  • 1
  • 4
  • 4
  • 1
    As Kato mentions below, this is a Forge issue, not a Firebase issue. Try navigating directly to a deeper path to handle larger datasets. – Andrew Lee Apr 26 '13 at 21:19

1 Answers1

76

It's simply the limitations of the Forge UI. It's still fairly rudimentary.

The real-time functions in Firebase are not only suited for, but designed for large data sets. The fact that records stream in real-time is perfect for this.

Performance is, as with any large data app, only as good as your implementation. So here are a few gotchas to keep in mind with large data sets.

DENORMALIZE, DENORMALIZE, DENORMALIZE

If a data set will be iterated, and its records can be counted in thousands, store it in its own path.

This is bad for iterating large data sets:

/users/uid
/users/uid/profile
/users/uid/chat_messages
/users/uid/groups
/users/uid/audit_record

This is good for iterating large data sets:

/user_profiles/uid
/user_chat_messages/uid
/user_groups/uid
/user_audit_records/uid

Avoid 'value' on large data sets

Use the child_added since value must load the entire record set to the client.

Watch for hidden value operations on children

When you call child_added, you are essentially calling value on every child record. So if those children contain large lists, they are going to have to load all that data to return. Thus, the DENORMALIZE section above.

Kato
  • 40,352
  • 6
  • 119
  • 149
  • Do you mean, if we just want to get everything about an user just like display a user profile page then we use `"This is bad for iterating large data sets"` methods, otherwise, if want to display a list of user example for admin use then we use `"this is good for iterating large data sets method"` ? – vzhen May 21 '13 at 01:29
  • 5
    @vzhen pretty much, the trick is remembering that when you call child_added or value (it doesn't matter which) that you are downloading all data under that path, including all children, and not just the root level element you asked for. – Kato May 21 '13 at 13:46
  • 5
    As Kato mentioned, child_added is named a little bit confusingly since it also returns all data under a path regardless of when it was actually added. If you want "event whenever a new child is added", you'll need to do something like `ref.limitToLast(1).on('child_added', ...`. – Cory Mawhorter May 04 '15 at 21:48
  • 1
    how `child_added` on `/users/uid/chat_messages` is different from `child_added` on `/user_chat_messages/uid`? I want to use the first approach but will never have any listener directly on `/users/uid`. – Zeeshan Ali Dec 21 '16 at 22:58
  • @ZeeshanAli Not sure if you found out already. But this will be slow when using the `/users/uid/chat_messages` setup and you want to fetch the user info to show on a profile page. It'll fetch ALL `chat_messages` along with it. Use different data paths, and connect based on an id. In short: the `This is bad for iterating large data sets`-approach will easily slow down your app, where the `This is good for iterating large data sets`-approach will make it very hard to slow down your app. – Guido Bouman Jun 07 '17 at 07:23
  • Is it realistic to expect to make 1000 database queries per request? About how long will that take? – pete Jun 13 '17 at 05:14