Index messed up if I upload more than one file at once

Question

I've got the following firebase function to run once a file is uploaded to firebase storage. It basically gets its URL and saves a reference to it in firestore. I need to save them in a way so that I can query them randomly from my client. Indexes seem to be to best fit this requirement.

for the firestore reference I need the following things:

1. doc ids must go from 0 to n (n beeing the index of the last document)
2. have a --stats-- doc keeping track of n (gets incremented every time a document is uploaded)

To achieve this I've written the following node.js script:

const incrementIndex = admin.firestore.FieldValue.increment(1);

export const image_from_storage_to_firestore = functions.storage
  .object()
  .onFinalize(async object => {
    const bucket = gcs.bucket(object.bucket);
    const filePath = object.name;
    const splittedPath = filePath!.split("/");

    // se siamo nelle immagini
    // path = emotions/$emotion/photos/$photographer/file.jpeg

    if (splittedPath[0] === "emotions" && splittedPath[2] === "photos") {
      const emotion = splittedPath[1];
      const photographer = splittedPath[3];
      const file = bucket.file(filePath!);

      const indexRef = admin.firestore().collection("images")
        .doc("emotions").collection(emotion).doc("--stats--");

      const index = await indexRef.get().then((doc) => {
        if (!doc.exists) {
          return 0;    
        } else {
          return doc.data()!.index;
        }
      });

      if (index === 0) {
        await admin.firestore().collection("images")
            .doc("emotions")
            .collection(emotion)
            .doc("--stats--")
            .set({index: 0});
      }

      console.log("(GOT INDEX): " + index);

      let imageURL;
      await file
        .getSignedUrl({
          action: "read",
          expires: "03-09-2491"
        })
        .then(signedUrls => {
          imageURL = signedUrls[0];
        });

      console.log("(GOT URL): " + imageURL);

      var docRef = admin.firestore()
        .collection("images")
        .doc("emotions")
        .collection(emotion)
        .doc(String(index));

      console.log("uploading...");


      await indexRef.update({index: incrementIndex});
      await docRef.set({ imageURL: imageURL, photographer: photographer });

      console.log("finished");
      return true;
    }
    return false;
  });

Getting to the problem:

It works perfectly if I upload the files one by one. It messes up the index if I upload more than one file at once, because two concurrent uploads will read the same index value from --stats-- and one will overwrite the other.

How would you solve this problem? would you use another approach instead of the indexed one?

Answer 1

You should use a Transaction in which you:

1. read the value of the index (from "--stats--" document),
2. write the new index and
3. write the value of the imageURL in the "emotion" doc.

See also the reference docs about transactions.

This way, if the index value is changed in the "--stats--" document while the Transaction is being executed, the Cloud Function can catch the Transaction failure and generates an error which finishes it.

In parallel, you will need to enable retries for this background Cloud Function, in order it is retried if the Transaction failed in a previous run.

See this documentation item https://firebase.google.com/docs/functions/retries, including the video from Doug Stevenson which is embedded in the doc.