1

Hi I read that the limitation of the Firestore max document size is 1MiB.

I want to store heart rate and other "activity" data and my current model is similar to this

-activity
  - HR points stream (1 point per second up to 86400 points)
  - Lat points stream (same as above)
  - Long points stream (same as above)

The above number (86400), for example, is for 24h of storage.

In general, I am trying to eg store and get a let's say 24h run that someone did.

However, due to the firestore limitations on size, this is getting to predict.

Can you suggest or guide to the right direction of what would someone do in order to store such "big" data?

I have tried to separate the above streams to their own documents but with no luck as a stream of heartrate even can get too much data for the firestorm to store.

So in short what are the recommendations for storing big streams of data only with the firestore and not using for example cloud storage?

Jimmy Kane
  • 16,223
  • 11
  • 86
  • 117

1 Answers1

5

Can you suggest or guide to the right direction of what would someone do in order to store such "big" data?

In order to store such "big" data you should change the way you are holding that data from within a single documents to a collection. In case of collections, there is no limitation. You can add as many documents as you want. According to the official documentation regarding Cloud Firestore Data model:

Cloud Firestore is optimized for storing large collections of small documents.

So you should take advantage of this feature.

For details, I recommend you see my answer from this post where I have explained some practices regarding storing data in arrays (documents), maps or collections.

Alex Mamo
  • 130,605
  • 17
  • 163
  • 193
  • 1
    this part : "As a conclusion, put data in the same document only if you need it to be display it together. Also don't make them so big so you'll need to download more data that you actually need. " That helps me a lot to think about how this should be structured. Thanks a lot Alex. – Jimmy Kane Dec 06 '18 at 13:30
  • I ll keep the question a bit open and will accept this (most probably) after a few (just to give a chance to other members, just in case) – Jimmy Kane Dec 06 '18 at 13:37
  • Small question: I read that if in a document there are subcollection as they grow they do not impact the size of the document. Perhaps having the document as the parent entity would help and after that model the streams? – Jimmy Kane Dec 06 '18 at 13:39
  • 1
    "I read that if in a document there are subcollection as they grow they do not impact the size of the document." That's correct, it won't. "Perhaps having the document as the parent entity would help and after that model the streams?" That's right. – Alex Mamo Dec 06 '18 at 13:40
  • 1
    Alright Jimmy, leave it open. If you also need other informations, don't hesitate to ask me. – Alex Mamo Dec 06 '18 at 13:41
  • Funny I cannot anyway fit my models in firebase except from using storage for the full data. Or if I structure my data chronological and then segment, also maintaining headers etc. Imagine perhaps (in a hard load scenario) that a user has eg 10 collections of activities, each activity has 10+ streams of 1s points. So in AVG each activity can have 10+ 4mb-20mb data points. :-( Makes sense? – Jimmy Kane Dec 06 '18 at 17:00
  • Loading data in "segments", it's always a good option. Yes, it's quite a lot but rememeber, in Firestore everything is about the number of reads and writes. So you should structure your database so it can be queried with less reads. – Alex Mamo Dec 06 '18 at 17:16
  • yeah I suppose and from what I see it's not so much a firestore issue but a more logical one. I was using LZ compression and localstorage and I wanted to avoid that because it does not scale. Perhaps Ill end up writing "header/summary" data to the Firestore and somehow try to format as of segmenting but it's hard it you still need all the data to plot it :-( Anyways thanks for the help! – Jimmy Kane Dec 06 '18 at 17:44
  • Wouldn't this scale terribly? This answer isn't necessarily wrong, but considering its a stream of data, I assume the data would generally all need to be read at once. Wont using a separate document read for each tiny piece of data significantly increase costs? – twiz May 21 '19 at 16:30
  • @twiz It won't scale as you say, it will scale perfectly fine as long as you are using collections. A documented is limited to 1Mib, so you cannot out much data in it. – Alex Mamo May 21 '19 at 18:26
  • @AlexMamo I meant that the cost would increase significantly because it would require a "read" for every document in that collection. Or am I misunderstanding Firestore pricing? – twiz May 25 '19 at 19:50
  • @twiz If your data doesn't fit in a document, your only chance is to use collections. And yes, you'll be charged for every document read. – Alex Mamo May 26 '19 at 06:25
  • @AlexMamo I understand your point, but in some cases I think you may be better off saving it as an array or even multiple arrays spread across collections, simply because it would avoid excessive reads. Of course, that obviously isn't a feature of Firebase and you'd have to implement the logic to pull the data back together, so it's definitely a tradeoff. – twiz Jun 02 '19 at 22:59