0

Here is my sample code that is making many parallel writes to Firestore.

const admin = require('firebase-admin');

admin.initializeApp({credential: admin.credential.applicationDefault()});
const db = admin.firestore();

const main = async () => {
    // Gather up documents and write to new app subcollection
    startTimeMs = Date.now();
    let commits = [];
    let doc = {abc: "def"};
    let total = 10;
    for (let i=0; i<total; i++) {
        commits.push(db.collection(`/test`).add(doc));
    }
    const bar = await Promise.all(commits);
    console.log(bar);
    const count = await db.collection(`/test`).count().get();
    console.log("count", count);
};

(async () => {
    await main();
})();

I tried total=10, 100, 1000.

When total is small (< 500), this code resolves quickly. However when total>500, this code takes a long time to finish.

In both cases, console.log("count", count); gets executed fairly quickly and shows the right number of documents (doesn't seem like some promises are failing). But when total > 500, the main function takes a lot longer to finish running.

I was expecting for all cases, the function would finish after executing the last line.

Jeff Huang
  • 1
  • 1
  • It is very likely that you are exceeding Firestore limitations when writing over 500 documents in parallel https://firebase.google.com/docs/firestore/quotas – TBA Jan 11 '23 at 19:20
  • To add onto the previous comment. When you exceed certain cloud computing limits, they may rate limit you and purposely slow down your operation. It is pretty much never a good idea to run an arbitrarily large number of parallel asynchronous operations either because of rate limiting or just poor efficiency for trying to run so many requests at the same time. You can use something like [mapConcurrent()](https://stackoverflow.com/questions/46654265/promise-all-consumes-all-my-ram/46654592#46654592) to run N at a time where you set what N is. – jfriend00 Jan 11 '23 at 19:34
  • Also, `(async () => { await main(); })();` isn't doing anything useful. You can replace it with `main().then(() => console.log("all done").catch(err => console.log(err))`. This will let you know precisely when it's done and will log any error you get. – jfriend00 Jan 11 '23 at 19:36
  • You should use [`db.batch`](https://firebase.google.com/docs/reference/node/firebase.firestore.Firestore#batch) instead of opening 1 connection for each `add` operation – Manuel Spigolon Jan 13 '23 at 16:58

0 Answers0