Meteor / Node.js: Multiple http requests within a for loop, evenly across a timed interval?

Question

Dear fellow JS developers -

I'm trying to make a batch of server-side http requests within a Meteor app on a timed interval and make updates to the MongoDB database based on the responses received. Every 15 seconds, the batch of requests should begin and the set of requests should be spread evenly across the 15 seconds (rather than flooding the API server all at once). To spread the requests evenly, I am attempting to use the setTimeout method.

There are two parameters that change between each http request, and these varying values are stored in two separate arrays. Rather than spelling out each and every http request (if you count all the combinations between the two arrays, 20*4=80), I've written a for loop within a for loop to greatly consolidate the code. For each response, if it comes back OK, two switch statements handle how the response should be treated by evaluating it's i and y values.

The problem: I seem to be in "callback hell". When the responses arrive from the server, the values of i and/or y sometimes have already been incremented by the loop system, so I cannot with 100% certainty handle the responses with the switch statements. The updatedb() function ends up performing certain calculations on the wrong responses (storing them in the wrong places in the database).

Hoping someone can provide some guidance on what I can do differently to fix this as I'm at my wit's end.

P.S. I tried doing this with a recursive approach but got a Maximum call stack size exceeded error.

---

test = function test(){
  // API base URL
  var baseURL = "https://example.com/";
  // Array1
  var array1 =  ['item1', // i = 0
                 'item2', // i = 1
                 'item3', // i = 2
                 'item4', // i = 3
                 'item5', // i = 4
                 'item6', // i = 5
                 'item7', // i = 6
                 'item8', // i = 7
                 'item9', // i = 8
                 'item10', // i = 9
                 'item11', // i = 10
                 'item12', // i = 11
                 'item13', // i = 12
                 'item14', // i = 13
                 'item15', // i = 14
                 'item16', // i = 15
                 'item17', // i = 16
                 'item18', // i = 17
                 'item19', // i = 18
                 'item20']; // i = 19
  // Array2
  var array2 = ['/path1/',   // y=0
                '/path2/',   // y=1
                '/path3/',   // y=2
                '/path4/'];  // y=3

  var count = 1;
  var timeout = Math.round(interval/(array1.length*array2.length)*count);

  // Iterate over each item in array1
  Meteor.setTimeout(function() {
    for (i=0;i<array1.length;i++) {
      // Iterate over each path in array2
      for (y=0;y<array2.length;y++) {
        var request = Meteor.http.call("GET", baseURL + array1[i] + array2[y]);
        // If response is OK, then:
        if (request.statusCode == 200) {
          // do meaningful things
          function updatedb(value) {
            switch (y) {
              case 0: /*do something with uniqueValue for case of y=0 */; break;
              // case 1, case 2, case 3
            }
          }
          switch(i) {
            case 0: updatedb(uniqueValue); break;
            // case 1, case 2, case 3, case 4, case 5...
          }
        } else {
          throw new Meteor.Error(500, "API call failed with error: " + request.status_txt);
        }
      }
    }
  }, timeout);
  count++;
}

var interval = 15000;
Meteor.setInterval(function(){
  test();
}, interval);

Answer 1

for (i=0;i<array1.length;i++)

This is one of deadly sins in Javascript. You declared a global variable i and iterate over that global variable. That way EVERY loop that iterates over i will iterate over THE SAME variable. Not what you want I believe.

Rule of thumb:  Always declare variables with var keyword. 

(Exception: intentionally creating global variables.)

for (var i=0; i<array1.length; ++i)

Second problem with your code is more tricky, but fortunately well-known. Look at this code:

var funs = [];

for(var y=0; y<10; ++y) {
  var print = function() {
    console.log(y);
  }
  funs.push(print);
}

for(var i=0; i<funs.length; ++i) {
  funs[i]();
}

What would you expect is printed to console when you run the code? Think about it. Then check it out: http://repl.it/Uzb.

The second warning you see at that page says it all:  Don't make functions within a loop . What happens is that variable y you used inside the function body refers to the variable y scoped outside of the for block, which was already incremented to 10 by the time each of the function you've defined has been called. So console.log(y) always resolves to console.log(10).

Rule of thumb:  Don't make functions within a loop. 

Answer 2

You can use an event on the element that gets updated after the call.

For an example you can look at http://test-this.ro/assign-the-same-tests-to-the-zephyr-suites-after-they-are-moved-to-another-project/ line #19 "$('body').on('requestscompleted', function(){..."