In reference to this answer large number of promises of how to handle a high number of queries I have regrouped the queries as shown using the 'lodash' library which works for a low number of queries however firebase returns an error
@firebase/database: FIREBASE WARNING: Exception was thrown by user callback. RangeError: Maximum call stack size exceeded
Which I know means that the arrays have grown too large however, when I try running pure Javascript Promises with a 10 ms timer the code seems to hold up to 1,000,000 as shown in that answer. I am not sure if this is a firebase or a node.js issue but given that firebase real time database can store millions of records in a JSON tree there must be a better way to process so many promises. I have largely based the approach off of these three questions, this was the original problem Find element nodes contained in another node , this approach for checking the database which requires so many reads check if data exists in firebase, and this approach for speeding up the requests Speed up fetching posts for my social network app by using query instead of observing a single event repeatedly
I am not sure if I am performing all of these reads correctly especially since it is such a high volume, thank you.
exports.postMadeByFriend = functions.https.onCall(async (data,context) => {
const mainUserID = "hJwyTHpoxuMmcJvyR6ULbiVkqzH3";
const follwerID = "Rr3ePJc41CTytOB18puGl4LRN1R2"
const otherUserID = "q2f7RFwZFoMRjsvxx8k5ryNY3Pk2"
var promises = [];
console.log("start")
var refs = [];
for(var x = 0; x < 100000; x +=1){
if (x === 999){
const ref = admin.database().ref(`Followers`).child(mainUserID).child(follwerID)
refs.push(ref);
continue;
}
const ref = admin.database().ref(`Followers`).child(mainUserID).child(otherUserID);
refs.push(ref);
}
function runQuery(ref){
return ref.once('value');
}
const batches = _.chunk(refs, 10000);
refs = [];
const results = [];
while (batches.length) {
const batch = batches.shift();
const result = await Promise.all(batch.map(runQuery));
results.push(result)
}
_.flatten(results);
console.log("results: " + JSON.stringify(results));
})