Firestore with data higher than 1Mb?

Question

I'm using Firebase to store articles written by my users, and these articles might be more than 1 MiB of content.

I know that Firestore limits the document size to 1 MiB and I'm trying to figure out a way to work around that.

I saw that post on Stackoverflow but it doesn't apply here because it's related to a list of items being too big.

My initial idea was to store the content in Cloud Storage instead, and, when loading the article document, if content starts with gs://, I would load the document from the storage, and set it to "content" directly, but I'm afraid this will be too slow.

How can I implement a way to store more than 1 MiB content in Firestore?

Personally I find your base solution pretty good, I've already done the same thing on pages that had to load a dozen of images and videos each. There was actually no noticeable latency that would annoy me. Just simply stored the GS links in my documents so I could easily request on them asynchronously front-side. — Ryo Shiina, Nov 23 '21 at 18:07
Thank you @RyoShiina. It seems you have more experiences than me :) From the doc,it says that you can create a reference from a Storage. Do you think that "reference" would work on a Firestore document directly? — Cyril N., Nov 23 '21 at 18:09
I'm afraid I don't have that much experience, I only had 3-4 small projects with Firestore. But I've never heard or read about direct reference between Firestore and GS. I probably overlooked that part of the docs, sorry! — Ryo Shiina, Nov 23 '21 at 18:16
@RyoShiina No worries. If that doesn't exists, they should add it :D it would be awesome — Cyril N., Nov 23 '21 at 19:26

score 2 · Answer 1 · answered Nov 23 '21 at 18:09

My rule of thumb is to use Firestore only for the structured data that you want to query in a use-case such as yours.

The other data I'd indeed put into Cloud Storage files that you then associate with the document (e.g. using the same ID, or by storing the path in a field).

If you need to listen to realtime updates to the data in the Storage file, update a field in the document (e.g. lastUpdated) after you write an update to Storage - to notify the listeners that they need to reload the data from Storage too.

The only alternative I've used is to overflow data to documents with a sequential ID suffix (so just adding -1, -2, etc to the document ID), but that was only in cases where I want to query the document IDs (instead of fields in there) and need to have a realtime listener on the contents of the documents.

Alex Mamo · Accepted Answer · 2021-11-24T08:56:17.527

I personally answered that question, and yes, it refers to a list of items. In your case, if an article contains text as well as pictures, I'll indeed add all images to the Firebase Storage and the text to Firestore, and link them as already @FrankvanPuffelen mentioned in his answer. Besides that, as @ToddKerpelman states in the following answer:

firestore database add image to record

Don't add pictures to Firestore. Cloud Storage is meant for that.

Thinking about your use case, to get "over" that limitation, there is another (similar) approach. Instead of trying to save all the data in a single document, I'd add it to multiple documents like this:

Firestore-root
  |
  --- articles (collection)
        |
        --- $articleId (document)
             |
             --- articleParts (collection)
                    |
                    --- $firestoreRandomId
                    |      |
                    |      --- text: "Text of the article"
                    |      |
                    |      --- part: 1
                    |      |
                    |      --- articleId: $articleId
                    |
                    --- $firestoreRandomId
                           |
                           --- text: "Remaining text of the article"
                           |
                           --- part: 2
                           |
                           --- articleId: $articleId

And as you can see, each article consists of several parts. To get all the articles from the database you have to create a collectionGroup query, that look like this:

db.collectionGroup("articleParts");

To read each article individually, simply concatenate all the parts for each article according to the articleId and part number.

On the other side, if you only want to read a single article, create a reference that points to:

db.collection("allArticles").document($articleId).collection("articleParts");

And do the same operation as above.

Or you can use:

db.collectionGroup("articleParts").whereEqualTo("articleId", someId);

But here, don't also forget to create an index.

In this way, you can create as many documents as you want for a single article. For Android, I have created a library called FirestoreDocument-Android, which will help you always check the size of a document against the maximum of 1 MiB (1,048,576 bytes) quota. So if the article is bigger than the maximum limitation, split the article and add the content in multiple documents.

Firestore with data higher than 1Mb?

2 Answers2