Is there a way to batch read firebase documents

Question

I am making a mobile app using flutter with firebase as my backend.

I have a collection of user document that stores user information. one of the fields is an array of references (reference documents in another collection) which I want to use in an operation like batch that in that would then allow be to read all the documents.

I know batch only allows writes to the database, My second option would be Transaction, which requires writes after reads which I am trying to avoid.

Is there a way to read multiple documents in one operation without having to use Transaction?

Your last sentence isn't clear to me. Please edit the question to explain what exactly you're trying to accomplish with a batch that you would normally do with a transaction. It will be helpful if you give a specific example of document contents. — Doug Stevenson, Jan 03 '20 at 03:50
You can use an `IN` query to load up to 10 documents by their ID in one go. See https://stackoverflow.com/questions/46721517/google-firestore-how-to-get-document-by-multiple-ids-in-one-round-trip. If you need more than 10 documents, you'll have to do multiple queries, so you might as well simply load them individually. — Frank van Puffelen, Jan 03 '20 at 04:04
For those using Node, go to [Garret's answer](https://stackoverflow.com/a/70611213/2619537) below. For those using other clients, an option could be to implement a firebase function using Node Admin SDK to retrieve those document references and send them back to the client. Depends on your specific use case, of course. — maganap, Apr 18 '23 at 12:44

score 14 · Answer 1 · answered Jan 03 '20 at 04:11

14

Firestore doesn't offer a formal batch read API. As Frank mentions in his comment, there is a way to use IN to fetch multiple documents from a single collection using their IDs. However, all of the documents must be in the same collection, and you can't exceed 10 documents per query. You might as well just get() for each document individually, as the IN query has limitations, and isn't guaranteed to execute any faster than the individual gets. Neither solution is guaranteed to be "consistent", so any one of the documents fetched could be "more fresh" than the others at any given moment in time.

answered Jan 03 '20 at 04:11

Doug Stevenson

297,357
32
422
441

1

Thanks for the answer Doug, Although I am trying to get multiple documents from different collect so IN wont necessarily solve my problem. Your answer help me understand the capabilities and restriction of firebase. – William Jan 03 '20 at 04:50
4

@doug-stevenson "and you can't exceed 10 documents per query", the doc only mentiones that "Use the IN operator to combine up to 10 equality (==) clauses on the same field " from my understanding, the limitation is that you can query up to 10 equality clauses on one field and get back as many documents as there are. not only 10 documents, am I wrong? – Yacine Benali Aug 13 '20 at 15:39
1

@Yachine true, but in this case he's performing an IN query on the id attribute. So it's the same thing here. – Noam Sep 16 '20 at 19:01
There is a benefit to using IN over using serial gets, if you are sorting by a field and have a limit on the number of docs you want to be returned. With the IN query those may all come from a single field value as it may have many more documents and it fills up the 'limit', whereas when doing it via concurrent gets you would have to apply the limit to each get (resulting in up to 10x as many total reads) and then sort on your end, and then truncate if desired. – Albert Renshaw Feb 23 '21 at 01:07

score 5 · Answer 2 · answered Jan 06 '22 at 17:35

If you know the document IDs and the collection paths of the documents needed to be fetched, you could always use the getAll() method which is exposed in the firebase Admin SDK (at least for Node.js environments).

Then, for example, you could write an HTTPS Callable Function that would accept a list of absolute document paths and perform a "batch get" operation on them using the getAll() method.

e.g.

// Import firebase functionality
const functions = require('firebase-functions');
const admin = require('firebase-admin');

// Configure firebase app
admin.initializeApp(functions.config().firebase);

// HTTPS callable function
exports.getDocs = functions.https.onCall((data, context) => {

    const docPathList = data.list; // e.g. ["users/Jkd94kdmdks", "users/8nkdjsld", etc...]
    const firestore = admin.firestore();

    var docList = [];

    for (var i = 0; i <= docPathList.length - 1; i++) {

        const docPath = docPathList[i];
        const doc = firestore.doc(docPath);
        docList.push(doc);
    }

    // Get all
    return firestore.getAll(...docList)
        .then(results => {
            return { data : results.map(doc => doc.data()) };
        })
        .catch(err => {
            return { error : err };
        })
});

Not sure what the limit (if any) is for the number of documents you can fetch using getAll(), but I do know my application is able to fetch at least 50 documents per call successfully using this method.

I just used `getAll()` to query for 9344 document refs... it took only 6 seconds (it will depend on your document data, though). I also used spread operator for the arguments (like `firestore.getAll(...docList)`), so the function took 9344 arguments. Smoothly went through. Node 16 on a deployed firebase function. — maganap, Apr 18 '23 at 12:21

score 4 · Answer 3 · answered Jan 20 '21 at 17:19

4

Firestore has a REST API that allows you to do batch GETs with document paths that may be what you need.

See https://firebase.google.com/docs/firestore/reference/rest/v1beta1/projects.databases.documents/batchGet

answered Jan 20 '21 at 17:19

Jacob Wright

311
2
10

Is there a way to batch read firebase documents

3 Answers3

Linked