How to determine once all async readFile() calls have completed

Question

I am looping over file directories and for each location I am reading the files in this directory with an async fs.readFile(). How can I best determine if all the async readFile() calls have completed?

Answer 1

The general strategy is to keep track of the number of files you have already read by incrementing a shared counter from fs.readFile callbacks. Then, when this counter equals the total number of files, you know you are done. For example:

function readFiles(paths, callback) {
  var processed = 0,
      total = paths.length;

  // holds results (file contents) in the same order as paths
  var results = new Array(total);

  // asynchronously read all files
  for (var i = 0, len = files.length; i < len; ++i) {
    fs.readFile(paths[i], doneFactory(i));
  }

  // factory for generating callbacks to fs.readFile
  // and closing over the index i
  function doneFactory(i) {
    // this is a callback to fs.readFile
    return function done(err, result) {
      // save the result under the correct index
      results[i] = result;

      // if we are all done, callback with all results
      if (++processed === total) {
        callback(results);
      }
    }
  }
}

which can be used like:

readFiles(['one.txt', 'two.txt'], function (results) {
  var oneTxtContents = results[0];
  var twoTxtContents = results[1];
});

---

If you are on Node 0.11.13 or higher, you can also make use of native promises, and in particular Promise.all, which takes an array of promises and waits for all of them to be resolved:

function readFiles(paths) {
  // for each path, generate a promise which is resolved
  // with the contents of the file when the file is read
  var promises = paths.map(function (path) {
    return new Promise(function (resolve, reject) {
      fs.readFile(path, function (err, result) {
        if (err) {
          reject(err);
        } else {
          resolve(result);
        }
      });
    });
  });

  return Promise.all(promises);
}

which can be used like:

readFiles(['one.txt', 'two.txt']).then(function (results) {
  var oneTxtContents = results[0];
  var twoTxtContents = results[1];
});

---

Finally, there are a number of libraries which make this (rather common) task easier.

For callback-based parallel asynchronous tasks, you can use the async library:

async.map(['one.txt', 'two.txt'], fs.readFile, function (err, results) {
  var oneTxtContents = results[0];
  var twoTxtContents = results[1];
});

For Promise-based parallel asynchronous tasks, you can use a Promise library like Q, which contains utilities for "promisification" of Node-style functions like fs.readFile:

// this is not the only way to achieve this with Q!
Q.all(['one.txt', 'two.txt'].map(function (path) {
  // call "promisified" version of fs.readFile, return promise
  return Q.nfcall(fs.readFile, path);
})).then(function (results) {
  // all promises have been resolved
  var oneTxtContents = results[0];
  var twoTxtContents = results[1];
});