Firebase for a "big" amount of data

Question

We are planning migrating our application to Firebase for realtime capabilities and to improve performance. It`s a sales application which stores costumers products and sales data. Our costumers (we call it Groups) have in general 10 stores each (we call it Companies).

It`s not a big deal to handle this amount of data. But recently we made a deal with a big Group which has 1.000+ stores (and 20 new ones every month).

My question is how to structure this data without having performance issues and keep the realtime features?

I think if we focus on how to structure Groups-Companies we can handle the other parts (sales, products, etc)

"Groups": {
    "Group1": {
      "name": "Historical Tech Pioneers"
    },
    "Group2": { ... }
}

"Stores": {
    "Store1": {
      "name": "abc",
      "group" "Group1"
    },
    "Store2": {
      "name": "def",
      "group" "Group1"
    },
}

Is this data structure ok to deal with thousands of records every time?

I am very concern about Firebase performance if every time I have to query for over 1.000 stores. Everybody says that 1.000 records is nothing for Firebase, but I did some tests and found that retrieving 1.000 records is not as fluid as they say.

You want a query to retrieve all 1000 records to some client app? That doesn't sounds like a good idea. Typically you want to use an index query by some value in those records to retrieve just one or a small subset of them. — Doug Stevenson, Jul 22 '17 at 19:41
[https://firebase.google.com/docs/database/web/structure-data](https://firebase.google.com/docs/database/web/structure-data) — robbannn, Jul 22 '17 at 19:47
Actually, not all Users will access all Stores. I think a good idea will be a User_Stores node where I can store all Stores a user has access to. — Hugo Ramos, Jul 22 '17 at 20:56
@HugoRamos "I did some tests and found that retrieving 1.000 records is not as fluid as they say" Please show the code for your test. Better yet: reproduce it in a place where we can access it, such as a jsbin. Only then can we all look at the same thing and help you optimize. Also see my explanation of list size limits here: https://stackoverflow.com/questions/39712833/firebase-performance-how-many-children-per-node/39713060#39713060 — Frank van Puffelen, Jul 22 '17 at 20:57
Discussing NoSQL data models is only meaningful in the context of a specific use-case. You'll often expand your data model to cater for new use-cases. For a general introduction see [NoSQL data modeling](https://highlyscalable.wordpress.com/2012/03/01/nosql-data-modeling-techniques/) and view [Firebase for SQL developers](https://www.youtube.com/playlist?list=PLl-K7zZEsYLlP-k-RKFa7RyNPa9_wCH2s). — Frank van Puffelen, Jul 22 '17 at 21:01

score 0 · Answer 1 · edited Jul 22 '17 at 20:05

0

So i have na suggestion that is to organize the firebase data like that for exemple:

"Groups": {
    "Group1": {
        "name": "Historical Tech Pioneers"
        "Stores": {
            "Store1": {
                "name": "abc"
            },          
            "Store2": {
                "name": "def"
            }
        },
    },
    "Group2": { ... }
}

And to get the store just poit to Groups>Group1>Stores. For better performance add pages to firebase data, for exemple 10 stores per page.

Note: is only a suggestion.

edited Jul 22 '17 at 20:05

robbannn

5,001
1
32
47

answered Jul 22 '17 at 19:42

Bruno Ferreira

1,561
1
10
17

If one would want to fetch a list of all group names, it means you would get all of the groups stores. Thats a lot of unnecessary data being sent... – robbannn Jul 22 '17 at 19:59
Yeah you are right i dont think in that possibility i only think in the performance to get all stores, my bad. – Bruno Ferreira Jul 22 '17 at 20:04

robbannn · Answer 2 · 2017-07-22T20:50:27.857

0

Since fetching data at a location in your database also retrieves all of its child nodes, you want to keep the database structure as flat as possible. Your suggestion resembles the structure in the docs. Which to me seems your on the right track.

Looking at your example i would probably re-structure the Stores-node, to easily get all Stores of a group:

"Stores": {
    "Group1": {
        "Store1": {
            "name": "abc"
        },
        "Store2": {
            "name": "def"
        },
        ...
    },
    ...        
}

In the documentation (one example) i have read denormalization is a keyword. Don't be afraid to duplicate data in order to speed up your reads. How you intend to use your data should determine how you structure your data.

As Chris Esplin wrote:
'We have to constantly balance normalization (shallow structures) vs denormalization (deep structure) based on how we want to use our data.'

edited Jul 22 '17 at 20:50

answered Jul 22 '17 at 19:49

robbannn

5,001
1
32
47

Good to know we are doing it right. But the question remains: At client side, eventually, 1.000 records will be heavy to handle? – Hugo Ramos Jul 22 '17 at 20:58
Firebase is designed to handle large datasets. How do you handle the datasets client side? What event do you use to retrieve data? `value` `child_added`? – robbannn Jul 22 '17 at 21:13
I handle datasets on client side. I think one of the goals of using Firebase as BaaS is to handle data without the need of a custom API. Am I right? – Hugo Ramos Jul 23 '17 at 20:02

Firebase for a "big" amount of data

2 Answers2