Best practice: Angularfire database structure for 10,000+ strings

Question

What is the best practice for storage of 10,000+ strings in Firebase?

long_lists/
    <user1>/
        string1
        string2
        ...
        string15000
    <user2>/
        string1
        string2
        ...
        string15000

According to The Firebase Blog – Best Practices: Arrays in Firebase, arrays are a big no-no.

Right now, it looks like using the push() method to generate a unique key for every item (key1: string1, key2: string2, ...) makes the most sense:

firebase.database().ref('long_lists/<uid>').push().key

I understand the seemingly subjective nature of this question, but I'm guessing there are really only one or two ways to store long single-level lists for users in a high-performance way.

Storing data is hardly ever a problem. It's how you want to access the data, where the structure becomes more important and may become a limit on scalability. See my answer https://stackoverflow.com/questions/39712833/firebase-performance-how-many-children-per-node for some more — Frank van Puffelen, Aug 03 '17 at 03:50
@FrankvanPuffelen Great point, Frank – and the SO question you linked to was helpful. In my case, tho, there are no "recent" items I need access to. My application frequently examines the 10,000+ strings for validity, and potentially removes them. Do you think the {uniqueKey: string1, uniqueKey2: string2, ..} structure suitable for this? (see my comment on Michal's answer as well) — Drin, Aug 03 '17 at 14:50

score 2 · Accepted Answer · answered Aug 02 '17 at 20:21

2

TL;DR it depends...

I personally structure my data as a complex object:

firebaseKey: "abcde"
name: "test"
order: 0

And fetch them sorted according to order. When an object gets deleted I leave it as it is (still sorted) and when an object is added I increment the order of the last object (keeps it still sorted). Obviously when things need moving around I reshuffle all indices.

That being said, you did point out that your use-case is over 10k of strings per user. Well then, if they are not frequently modified or possibly even immutable, I'd do one long string separated by a unique character and do the heavy work on the client side.

firebaseKey: "abcde"
value: "string1#string2#string3#string4"

That way you're saving yourself space on firebase storage and letting the client do the work which can bring you even money savings down the line (less space in the cloud, less cloud power used).

answered Aug 02 '17 at 20:21

Michal

15,429
10
73
104

2

At first I did a double-take on that last snippet. Then I realized I had done the exact same thing recently on a big blob of opaque JSON. It's not a common practice, but there are use-cases where storing a "big blob of string" is the right solution. :-) – Frank van Puffelen Aug 03 '17 at 03:52
Thanks for the answer, @Michal! The "big blob" technique is what I was considering, as well. I only have one concern about it – because my application accesses the list frequently, parsing and stringifying the giant (300kb+) string all the time makes me nervous. But now that Frank saw your post and gave it the *O.K.* I'm leaning toward the "big blob" technique. – Drin Aug 03 '17 at 14:57
1

@Tread that is really huge...maybe consider pagination? Are all the strings unique for each user? Can you have a lookup table? This sounds a bit strange architecture-wise... – Michal Aug 03 '17 at 15:18
1

Accessing the list != Modifying the list. Access and parsing is fine. Frequent write operations would be quite expensive. – Michal Aug 03 '17 at 15:19
1

Just ran some tests... My old iPhone 6 can parse a JSON.parse a list of 20,000 strings (about 30 characters per string) in like 0.02 seconds haha! And I can JSON.stringify a 20,000 item Array AND have Firebase updated in about 1.5 seconds (which is like 1+ seconds faster than sending 20,000 nodes with unique ids to firebase). Unless my numbers are really off, **a single node with all of the strings concatenated** (JSON stringifyed in my case) **seems more than fast enough**. Guess computers are reaaalllll good at parsing data ;] thanks, again @Michal – Drin Aug 03 '17 at 16:15

Best practice: Angularfire database structure for 10,000+ strings

1 Answers1