I am having an issue where Firestore transactions freeze when run concurrently. As background, I have a Firestore collection, where each document has a dollar amount and a time. I am creating a function that releases a desired dollar amount from this collection, starting with the oldest documents.
For example, a function call to release $150 will iterate through the collection, removing dollar amounts from the collection until a total of $150 is removed. I do this using a recursive function which 1) finds the oldest dollar amount, 2) removes the inputted amount (i.e. $150) from that number, or deletes the number if the inputted amount is greater than the number, and 3) recurs if there is still a remaining amount to be removed. I use a Firestore transaction for step (2), as there is the possibility this collection will be changed by multiple users at the same time (and note that if I combine (1) and (2) to include the query in the transaction, the code behavior is unchanged).
The code below updates the collection correctly. However, it takes an extremely long time if it is called while an earlier instance is already running: if I call it once, then call it again before the first call is completed, it freezes and has potential to take 20-30 minutes instead of the usual 1-5 seconds. While it is clear that contention between concurrent transactions is causing the freezing (when I remove the write portion of the transaction, there is no freezing), the unknown question is what specifically about the contention is causing the freezing and how to fix it.
Addition: it seems this freezing may be related to https://github.com/firebase/firebase-tools/issues/2452. Consistent with that post, I am facing a 30 second freeze per transaction, which becomes many minutes given a single release has multiple transactions.
function releaseAmountFromStack(amount) {
return new Promise((resolve, reject) => {
let db = admin.firestore();
let stackRef = db.collection("stack");
stackRef.orderBy("expirationTime", "asc").limit(1)
.get().then((querySnapshot) => {
if(querySnapshot.empty) {
return reject("None left in stack");
}
let itemToRelease = querySnapshot.docs[0];
releaseItem(itemToRelease.ref, amount)
.then((actualReleaseAmount) => {
// If there is still more to release, trigger the next recursion
// If the full amount has been released, return it
if (amount > actualReleaseAmount) {
releaseAmountFromStack(amount-actualReleaseAmount)
.then((nextActualReleaseAmount) => {
return resolve(actualReleaseAmount + nextActualReleaseAmount);
})
.catch(() => {
return resolve(actualReleaseAmount);
});
} else {
return resolve(actualReleaseAmount);
}
});
});
});
}
function releaseItem(itemRef, amountToRelease) {
let db = admin.firestore();
return db.runTransaction((transaction) => {
return transaction.get(itemRef).then((itemDoc) => {
let itemAmount = itemDoc.data().amount;
let actualReleaseAmount = Math.min(amountToRelease, itemAmount);
// If item is exhausted, delete it. Else, update amount
if (actualReleaseAmount >= itemAmount) {
transaction.delete(itemDoc.ref);
} else {
transaction.set(itemDoc.ref, {
amount: admin.firestore.FieldValue.increment(-1*Number(actualReleaseAmount)),
}, {merge: true});
}
return actualReleaseAmount;
});
});
}
Here are some useful facts from the debug process so far. Thank you so much.
- During the freeze, it does not trigger a breakpoint on any of these lines of code. Only when the freeze finishes is a breakpoint triggered. This indicates the delay is not caused by looping through my code (if it were, a breakpoint should be triggered)
- The function eventually works as intended in that it releases the correct amount, it just takes a very long time. It will generally freeze, then execute, then freeze, then execute, and so forth until the process is completed
- Firestore usage stats show the function executes hundreds of reads and writes, even if it only needs to (and I would expect) it to iterate a few dozen times to release the requisite amount from the collection
