1

Below is a firebase read query performance test from this SO question, I just changed it to my own firebase instance and removed sequential test leaving only the parallel one.

The problem is that any time I run this code and I make the for loop i < 12 resulting in an array of length: 32,768, I keep getting an error saying that call stack exceeded: RangeError: Maximum call stack size exceeded but I can't figure out what is causing this error... I tried testing it on both localhost and via cloud functions but no success.

(The referenced SO test works at i<12 but then also breaks if I change it to 13).

function loadVideosParallel(videoIds, callback) {
  Promise.all(
    videoIds.map((id) => {
      return admin.database().ref("test").child(id).once('value')
        .then((snapshot) => {
          return snapshot;
        })
    })
  ).then((r) => callback());
}

function loadVideosForUser(loadVideosParallel, callback) {
  admin.database().ref("test").once('value').then((snapshot) => {
    var videoKeys = Object.keys(snapshot.val());

    for (var i=0; i < 13; i++) {
      videoKeys = videoKeys.concat(videoKeys);
    }

    loadVideosParallel(videoKeys, function() {
      if (callback) callback();
    });
  })
}

let start = Date.now();
loadVideosForUser(loadVideosParallel, function() {
  const endTime = (Date.now() - start)+'ms';
  console.log('parallel done after '+ endTime +'ms');
});

2017-12-20T23:26:28.527Z E helloWorld: RangeError: Maximum call stack size exceeded at T (/user_code/node_modules/firebase-admin/lib/database/database.js:139:366) at /user_code/node_modules/firebase-admin/lib/database/database.js:190:120 at ta (/user_code/node_modules/firebase-admin/lib/database/database.js:45:40) at ua (/user_code/node_modules/firebase-admin/lib/database/database.js:45:89) at Ag (/user_code/node_modules/firebase-admin/lib/database/database.js:190:93) at /user_code/node_modules/firebase-admin/lib/database/database.js:195:260 at Oc (/user_code/node_modules/firebase-admin/lib/database/database.js:85:363) at Oc (/user_code/node_modules/firebase-admin/lib/database/database.js:85:449) at Oc (/user_code/node_modules/firebase-admin/lib/database/database.js:85:449) at Nc (/user_code/node_modules/firebase-admin/lib/database/database.js:85:316) 2017-12-20T23:26:29.758741829Z D helloWorld: Function execution took 5107 ms, finished with status: 'crash'

Anybody have any insight? I tried looking at the /user_code/node_modules/firebase-admin/lib/database/database.js file but it's minified so can't really tell what's going on.

linasmnew
  • 3,907
  • 2
  • 20
  • 33

1 Answers1

2

You'r queueing up more than 32,768 read operations, which is overflowing one of the stacks in the SDK. Loading that many items without any flow control in your code seems like a bad idea, so I'm actually quite happy that the SDK cops out. Consider loading the items in batches that are more reasonable.

In general though: try to limit loading thousands of items from the database. If you need that many items for an aggregation operation, you're much better off keeping the aggregated value explicitly in the database and updating it with every write operation.

Frank van Puffelen
  • 565,676
  • 79
  • 828
  • 807
  • Thanks for the answer, but why do 32,768 read operations work in the referenced test, after stripping out the sequential part and only testing the parallel one? But if I copy it over to local host or my own cloud function it no longer works? – linasmnew Dec 21 '17 at 12:14
  • The motivation behind the test is that I'm trying to see how fanout-on-read vs fanout-on-write would differ for an activity feed – linasmnew Dec 21 '17 at 12:16