nodejs looping through array with async calls?

Question

I am trying to iterate through an array which pushes a new Thing to a list, inside the Thing it does some async calls of its own. How would I iterate through an array in a synchronous way, as the callback requires the the data from the list to work. Since my for loop is synchronous and does some asynchronous calls, the callback is called before the the list if finished being made.

I do not know how I would iterate through the array and do all the work before doing the callback

load(file, callback) {
  fs.readFile(file, (err, fd) => {
    var data = JSON.parse(fd);

    for(var i of data.array){
      this.list.push(new Thing(i.id)); // new Thing is Asynchronous
    }
    callback(); // Needs a finished list
    return;
  });
}

Solved it:

By converting my Thing class to synchronous, by removing the asynchronous calls to a function inside the class, and first instantiating all the Things in the loop then calling Promise.all calling the function I solved the issue:

load(file, callback) {
  fs.readFile(file, (err, fd) => {
    var data = JSON.parse(fd);

    for(var i of data.array){
      this.list.push(new Thing(i.id));
    }

    Promise.all(this.list.map(i => i.load()).then(callback);
  });
}

score 2 · Accepted Answer · answered Dec 26 '16 at 04:14

You'd have to have some state inside of Thing to track it's doneness for example you could have an instance variable that's a promise. So given this hacked together example of Thing

class Thing {
  constructor(id) {
    this.id = id;
    this.done = new Promise((resolve, reject) => {
      asyncThingWithCallback((err, data) {
        if (err) {
          this.asyncVal = null;
          reject(err);
        } else {
          this.asyncVal = data;
          resolve()
        }
      })
    });
  }
}

You can use the done property inside of your callback like this:

load(file, callback) {
  fs.readFile(file, (err, fd) => {
    var data = JSON.parse(fd);

    for(var i of data.array){
      this.list.push(new Thing(i.id)); // new Thing is Asynchronous
    }

    Promise.all(this.list.map((thing) => thing.done))
      .then(callback)
  });
}

score 2 · Answer 2 · edited May 23 '17 at 10:30

First off, it's generally not advisable to have a constructor that needs some asynchronous operation to finish creating a valid object. That just doesn't lead to easily writable or maintainable code because the constructor has to return the object reference and because the operation is asynchronous, it has to return that object reference before you're done creating a valid object. That just leads to messy, partially created objects. You can make it work by requiring a completion callback be passed to the constructor and making sure the calling code does not attempt to use the object until after the completion callback has been called, but this is just not a clean way to do things. It also makes it impossible to have your async operation return a promise (which is the future of async design) because the constructor has to return the object reference so it can't return a promise.

You could embed the promise in the object, but that's messy too because the promise is only really useful during the initial async operation.

What is often done instead is to make the constructor be only synchronous and then have a .init() method that does the async parts. That makes for cleaner code and is compatible with implementations using promises.

Or, you can create a factory function that returns a promise that resolves to the object reference.

Second off, as you already seem to know, your for loop runs synchronously. It doesn't "wait" for any async operations inside it to complete before going onto the next part of the loop. As long as each invocation of the loop is separate and doesn't depend upon the prior iteration, that's all fine. All you need to know is when all the async operations in the loop are done and making your async operations return promises and using Promise.all() is generally the best tool for that.

So, let's supposed you use the .init() method scheme where .init() does the async part of the initialization and the constructor is synchronous and .init() returns a promise. Then, you could do this:

// create all the things objects
let things = data.array.map(i => new Thing(i.id)); 

// initialize them all asynchronously
Promise.all(things.map(item => { 
    return item.init();
})).then(function() {
    // all things are asynchronously initialized here
});

Or, using the concept of a factory function that returns a promise that resolves to the object:

function newThing(i) {
    let o = new Thing(i.id);
    return o.init().then(function() {
        // resolve to the object itself
        return o;
    });
}

Promise.all(data.array.map(i => newThing(i))).then(things => {
    // all things in the array ready to be used here
});

If you need to sequence your array iteration so the 2nd iteration did not start until the async part of the first iteration was done and 3rd waited until the 2nd iteration was done and so on, then you can't use a for loop because it simply doesn't work that way. There are several different ways to do such a serialized async iteration. You can see several different schemes in these other posts:

How to synchronize a sequence of promises?

JavaScript: Perform a chain of promises synchronously

ES6 Promises - something like async.each?

How can I execute shell commands in sequence?

score -1 · Answer 3 · answered Dec 26 '16 at 06:50

You can use primise.all to run all the promises after the for loop .Then you can resolve the promise.all .

load(file) {
 fs.readFile(file).Then(function (fd){
var data = JSON.parse(fd);
var EachPromise = [ ]; 
for(var i of data.array){
EachPromise.push(new Thing(i.id)); // new Thing is Asynchronous
}
Promise.all(EachPromise)  .then(function (result){
console.log('this is result',result);  
}).Catch(function (error){
console.log('this is error', error);
});
}

nodejs looping through array with async calls?

3 Answers3