Promise.map: Maximum call stack size exceeded

Question

Im using bluebirds Promise.map() method to run 100,000 firebase queries as shown below and the function takes about 10 seconds to run.. If I set the concurrency higher than 1000 then I receive the error

Maximum call stack size exceeded

Any ideas on how to fix this and also how to speed this up. It seems to me that perhaps Promise.map() may not be the right function to use or maybe I am mismanaging the memory some how. Any ideas thank you.

exports.postMadeByFriend = functions.https.onCall(async (data, context) => {
            const mainUserID = "hJwyTHpoxuMmcJvyR6ULbiVkqzH3";
            const follwerID = "Rr3ePJc41CTytOB18puGl4LRN1R2"
            const otherUserID = "q2f7RFwZFoMRjsvxx8k5ryNY3Pk2"

            var refs = [];

            for (var x = 0; x < 100000; x += 1) {

                if (x === 999) {
                    const ref = admin.database().ref(`Followers`).child(mainUserID).child(follwerID)
                    refs.push(ref);
                    continue;
                }
                const ref = admin.database().ref(`Followers`).child(mainUserID).child(otherUserID);
                refs.push(ref);
            }

            await Promise.map(refs, (ref) => {
                return ref.once('value')
            }, {
                concurrency: 10000
            }).then((val) => {
                console.log("Something happened: " + JSON.stringify(val));
                return val;
            }).catch((error) => {
                console.log("an error occured: " + error);
                return error;
            })

Edits

const runtimeOpts = {
    timeoutSeconds: 300,
    memory: '2GB'
  }

exports.postMadeByFriend = functions.runWith(runtimeOpts).https.onCall(async (data, context) => {
            const mainUserID = "hJwyTHpoxuMmcJvyR6ULbiVkqzH3";
            const follwerID = "Rr3ePJc41CTytOB18puGl4LRN1R2"
            const otherUserID = "q2f7RFwZFoMRjsvxx8k5ryNY3Pk2"

            var refs = [];

            for (var x = 0; x < 100000; x += 1) {

                if (x === 999) {
                    const ref = admin.database().ref(`Followers`).child(mainUserID).child(follwerID)
                    refs.push(ref);
                    continue;
                }
                const ref = admin.database().ref(`Followers`).child(mainUserID).child(otherUserID);
                refs.push(ref);
            }

            await Promise.map(refs, (ref) => {
                return ref.once('value')
            }, {
                concurrency: 10000
            }).then((val) => {
                console.log("Something happened: " + JSON.stringify(val));
                return val;
            }).catch((error) => {
                console.log("an error occured: " + error);
                return error;
            })

Do you really need to do those 100.000 queries ? It seems that you are looking to show a user the posts that were made by a friend. You could perhaps use an infinite scroll system that will load 100 posts and then when the user reaches the end of the pages it will automatically loads the 100 next one — Weedoze, Jan 02 '19 at 15:00
You are retrieving 100.000 times the same `mainUserID` - `follwerID` combination. This makes no sense.. — R Pelzer, Jan 02 '19 at 15:09
Do you even get a better performance with that high number of `1000` compared to e.g. `500`. Higher `concurrency` does not mean better performance, especially as the bottleneck is most certainly the network connection (or the connection to the API in general). — t.niese, Jan 02 '19 at 15:20
Answer: _Don't_. You are making concurrent calls to a third-party API, a concurrency limit of somewhere in the 16-50 range is the absolute max. You're just asking to be rate limited. I would investigate (a) whether you need these queries at all, (b) if you need them, do them somewhere in a background worker where speed is not as important. — Elliot Nelson, Jan 02 '19 at 16:15
@Weedoze Unforunately this is the only solution I could come up with based on this asnwer https://stackoverflow.com/questions/53952903/how-do-i-efficiently-find-if-one-set-of-nodes-has-elements-contained-in-another, in essence the user selects a tag and from a tag I want to show their followers posts above the other posts pertaining to a tag. The best way I could come up with is to check each friend against a list of users who posted to a specific tag — TheRedCamaro3.0 3.0, Jan 02 '19 at 16:21
@RPelzer This was a mock up of a larger instance to test if promise.map would work for my case. In production the main userID would remain constant while checking against a list of followerIDs. As described in the comment above the solution is based on a user selects a tag and then checking to see if the displayed posts that fall under this tag were made by a user the mainUserID follows so as to show that post before the rest. It is a solution to this https://stackoverflow.com/questions/53952903/how-do-i-efficiently-find-if-one-set-of-nodes-has-elements-contained-in-another — TheRedCamaro3.0 3.0, Jan 02 '19 at 16:26
@t.niese surprisingly the performance does improve from 100 to 1000 from 49 seconds run time to 10 seconds run time and then just fizzles out with the error of maximum call stack size exceeded. — TheRedCamaro3.0 3.0, Jan 02 '19 at 16:29
@ElliotNelson would your recommend using the web worker library https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Using_web_workers. I haven't seen many examples combining web workers and promises Ive seen mostly comparisons of performance. How would one go about implementing this approach. — TheRedCamaro3.0 3.0, Jan 02 '19 at 16:31

Andrii Rudavko · Answer 1 · 2019-01-02T18:43:12.837

-1

Update: If the goal is to have a number of friends posts the better way to do it would be to have a cloud function that increments a counter on every new post saved to the DB. That way you can get the number of posts with no calculation needed.

Here is the similar answer, and a code sample with the like counter

Original answer: You could try to increase the memory allocated to your Cloud Funciton:

In the Google Cloud Platform Console, select Cloud Functions from the left menu.
Select a function by clicking on its name in the functions list.
Click the Edit icon in the top menu.
Select a memory allocation from the drop-down menu labeled Memory allocated.
Click Save to update the function.

As described in the Manage functions deployment page

edited Jan 02 '19 at 18:43

answered Jan 02 '19 at 15:34

Andrii Rudavko

370
1
7

Thank you for your answer, I tried upping the maximum memory allocated to 2GB but still I receive the same error. I am running the function locally I am not sure if that effects the maximum amount allocated as well. – TheRedCamaro3.0 3.0 Jan 02 '19 at 16:54
Thank you for your update, I appreciate your taking the time to help solve this. To implement a counter to record the number of posts a friend makes to a tag would require writes directly proportional to the number of friends a user has. Each time a user makes a post I would have to increment the counter on each node of their followers. – TheRedCamaro3.0 3.0 Jan 03 '19 at 03:23
That is right. Naturally, the solution is highly dependent on your systems architecture. But from the general point of view: number of new posts * user friends is much less then number of view times. Basically you should expect to have more page views than new posts. By shifting the intesive operations away to write time you allow a smoth reading experience for the user. – Andrii Rudavko Jan 04 '19 at 10:35

Promise.map: Maximum call stack size exceeded

1 Answers1