How do I return the accumulated results of multiple (parallel) asynchronous function calls in a loop?

Question

I have a function foo which makes multiple (parallel) asynchronous calls in a loop. I need to somehow wait until the results of all of the calls are available. How can I return the full results from foo, or otherwise trigger some processing after all of the data is available?

I tried adding each result to an array, but then the array isn't populated until after the point where I need to use it.

function foo() {
    var results = [];

    for (var i = 0; i < 10; i++) {
      someAsyncFunction({someParam:i}, function callback(data) {
        results.push(data);
      });
    }
    return results;
}

var result = foo(); // It always ends up being an empty array at this point.

Note: this question is deliberately generic along the lines of the existing generic "How do I return the response from an asynchronous call?" question. That question has some excellent answers, but doesn't cover multiple async calls. There are some other questions that mention multiple calls, but I couldn't find any loop-based ones, and some only had jQuery answers, etc. I'm hoping here for some generic techniques that don't depend on a particular library.

Answer 1

Use promises. Precisely, Promise.all was designed for this.

It takes an array (or iterable) of promises, and returns a new promise which is resolved when all the promises of the array have been resolved. Otherwise, it rejects when any promise of the array rejects.

function someAsyncFunction(data, resolve, reject) {
  setTimeout(function() {
    if(Math.random() < .05) {
      // Suppose something failed
      reject('Error while processing ' + data.someParam);
    } else {
      // Suppose the current async work completed succesfully
      resolve(data.someParam);
    }
  }, Math.random() * 1000);
}

function foo() {
  
  // Create an array of promises
  var promises = [];
  
  for (var i = 0; i < 10; i++) {
    // Fill the array with promises which initiate some async work
    promises.push(new Promise(function(resolve, reject) {
      someAsyncFunction({someParam:i}, resolve, reject);
    }));
  }
  
  // Return a Promise.all promise of the array
  return Promise.all(promises);
}

var result = foo().then(function(results) {
  console.log('All async calls completed successfully:');
  console.log(' --> ', JSON.stringify(results));
}, function(reason) {
  console.log('Some async call failed:');
  console.log(' --> ', reason);
});

Note that the results will be given according to the order of the array of promises, not in the order that the promises were resolved in.

Answer 2

A simple way of doing it would be to trigger a callback once all responses are in the array:

function foo(cb) {
    var results = [];

    for (var i = 0; i < 10; i++) {
      someAsyncFunction({someParam:i}, function callback(data) {
        results.push(data);

        if(results.length===10){
          cb(results);
        }
      });
    }

}

foo(function(resultArr){
    // do whatever with array of results
});

Only difference from the Promise.all approach is the order of the results is not guaranteed; but that is easily achievable with a few additions.

Answer 3

A long time ago I've answered a very similar question here: Coordinating parallel execution in node.js.

However, times have moved on. Since then a really good library have appeared and the promise design pattern have been fully explored and even standardized into the langauge. If you want to see how it can be done with raw code click the link above. If you just want to code read on..

async.js

The async.js library have basically implemented the code in the link above. With async the code you'd write would look something like this:

var listOfAsyncFunctions = [];

for (var i = 0; i < 10; i++) {
    (function(n){
        // Construct an array of async functions with the expected
        // function signature (one argument that is the callback).
        listOfAsyncFunctions.push(function(callback){
            // Note: async expects the first argument to callback to be an error
            someAsyncFunction({someParam:n}, function (data) {
                callback(null,data);
            });
        })
    })(i); // IIFE to break the closure
}

// Note that at this point you haven't called the async functions.
// Pass the array to async.js and let it call them.

async.parallel(listOfAsyncFunctions,function (err,result) {
    console.log(result); // result will be the same order as listOfAsyncFunctions
});

However, the authors of async.js have done more than that. Async also have functional array-like operations: each, map, filter, reduce. It makes asynchronously processing arrays simple and makes the code easier to understand:

var listOfParams = [];

for (var i = 0; i < 10; i++) {
    // Construct an array of params:
    listOfParams.push({someParam:i});
}

async.map(listOfParams,someAsyncFunction,function (err,result) {
    console.log(result);
});

Another thing async gives you is different algorithms for how to process the asynchronous tasks. Say for example you want to scrape a website but don't want them to ban your IP address for spamming their server. You can use async.series() instead of parallel to process the tasks one at a time:

// Set-up listOfAsyncFunctions as above

async.series(listOfAsyncFunctions,function (err,result) {
    console.log(result); // result will be the same order as listOfAsyncFunctions
});

Or if you want to process 3 tasks at a time:

async. parallelLimit(listOfAsyncFunctions, 3, function (err,result) {
    console.log(result); // result will be the same order as listOfAsyncFunctions
});

Promise.all()

The Promise.all() method works in a similar way to async.parallel() only it works with promises instead. You construct an array of promises then pass them to Promise.all():

var listOfPromises = [];

for (var i = 0; i < 10; i++) {
    // Construct an array of promises
    listOfPromises.push(somePromiseFunction({someParam:i}));
}

Promise.all(listOfPromises).then(function(result){
    console.log(result);
});

Answer 4

Don't use Promise.all! That fails the entire operation if any one of your promises fails!

Unless you're okay with that prospect, you'd be way better off doing something like this:

function sleep(ms) {
  return new Promise((resolve, reject) => {
    console.log(`starting ${ms}`);
    setTimeout(() => {
      if (ms > 1000) {
        console.log(`Threw out ${ms} because it took too long!`);
        reject(ms);
      } else {
        console.log(`done ${ms}`);
        resolve(ms);
      }
    }, ms);
  });
}

(async () => {
  console.log('aPromise, bPromise, cPromise executed concurrently as promises are in an array');
  const start = new Date();
  const aPromise = sleep(2000);
  const bPromise = sleep(500);
  const cPromise = sleep(5);
  
  try {
    const [a, b, c] = [await aPromise, await bPromise, await cPromise];
    // The code below this line will only run when all 3 promises are fulfilled:
    console.log(`slept well - got ${a} ${b} ${c} in ${new Date()-start}ms`);
  } catch (err) {
    console.log(`slept rough in ${err}ms`);
  }
})();

Answer 5

As other answers mentioned, Promises are a good way to go. Promise.all() was mentioned in one but that one returns and rejects immediately if one of the promises fails.

Promise.allSettled() is a good option if you want it to return only when ALL the promises are completed. This allows for handling if some promises were fulfilled while others were rejected.

Here's a sample from the Mozilla docs:

Promise.allSettled([
  Promise.resolve(33),
  new Promise(resolve => setTimeout(() => resolve(66), 0)),
  99,
  Promise.reject(new Error('an error'))
])
.then(values => console.log(values));

// [
//   {status: "fulfilled", value: 33},
//   {status: "fulfilled", value: 66},
//   {status: "fulfilled", value: 99},
//   {status: "rejected",  reason: Error: an error}
// ]