Node.js, async module, concurrency

Question

Am I best to avoid using more than one instance of an async module feature at a time?

My code has three parts to be performed in sequence. I'm using code like

var async = require('async');
async.waterfall(
  [ function(cb) { part1(); },
    function(cb) { part2(); },
    function(cb) { part3(); }
  ],
  function(err, res) { console.log('Done'); }
);

part1() has three parts, too. The third of those runs after the first two. My thought was to use

function part1(err, res) {
  async.waterfall(
    [ function(cb) {
        async.parallel(
          [ function(cb) { part1a(); },
            function(cb) { part1b(); }
          ]
        );
      },
      function(cb) { part1c(); },
      function(cb) {
        if (err) return err('part 1 error')
        return res()
      }
    ]
  );
}

Where part1a and part1b are running, I'm using three features of async: the outer waterfall, and the waterfall and parallel in part1. I restructured part1 to use only one feature: a waterfall, part1a then part1b then part1c just to get it working.

I am not achieving success. For example, I haven't seen a message 'Done'. And when I restructured part1, I get a call on part1a, but not on part1b or any other.

Maybe I'm not alone. In Optional callback not being called in the node.js async module's forEachOf method, @LuisDelgado indicated, "As a personal note, I have not had so far success in having async work nicely with a function that has an internal callback-async function itself."

In Complicated use case for Node.js Async Module, @Robbie and @dark_shadow found some success with async.auto. I guess that would involve flattening my code by breaking my three outer level functions into their components and calling each component from the outer level. Maybe that's "the JavaScript way"!

What's right?

Thanks in advance ...

EDIT: Nice job, Adam Sax. Thanks lots! A follow-on ...

EDIT: Adam and others suggested promises. Adam likes bluebird and showed some useful syntax. I'm having trouble with bluebird, appealing though it is. I posted a follow-on bluebird question here. With this edit, I'm removing that follow-on, leaving this question (as the title suggests) as an async question.

For those encouraging me to use promises, thanks! Please see node.js, bluebird, poor control of execution path.

The async question remains: Am I best to avoid using more than one instance of an async module feature at a time?

Answer 1

You may want to try using Promises for this sort of orchestration of asynchronous actions. Promises are now a native JS feature for nodejs and newer browsers. There are also a ton of libraries for Promises. One of the ones I use the most is bluebird which has some great utility functions and additional Promise sugar.

For your example you would have something like:

part1()
    .then(part2)
    .then(part3)
    .then(part4)
    .then(function() {
         console.log("Done!");
    });

function part1a() {
   return new Promise(function(resolve, reject){
     setTimeout(resolve, 1000);
   })
}

function part1() {
     return Promise.all([
         part1a(),
         part2b();
     ]);
}

Answer 2

I would also recommend using promises in this case because they are much nicer -- less code and a single chain of execution and error handling. If you want to stick with async, you need to call the callbacks passed to the async methods when your asynchronous operations are done.

var async = require('async');
async.waterfall(
  [ function(cb) { part1(cb); },
    function(cb) { part2(cb); },
    function(cb) { part3(cb); }
  ],
  function(err, res) { console.log('Done'); }
);

function part1(callbackFromAsync) {
  async.waterfall(
    [ function(waterfallCallback) {
        async.parallel(
          [ function(cb) { part1a(cb); },
            function(cb) { part1b(cb); }
          ], waterfallCallback // <- this needs to be called
        );
      },
      function(cb) { part1c(cb); },
      function(cb) {
          callbackFromAsync(); // <- this needs to be called!
      }
    ]
  );
}

Practically, each time an asynchronous operation is complete a cb function needs to be called to signal its completion. Imagine that part2, part3, part1a etc. are much simpler functions that look like this:

function part(callback) {
  someAsyncOperation(function (err, result) {
    callback(err, result);
  });
}

You have to call the callback passed into that function to signal the async operation is complete. You use the callback to pass the error (if any) and the result (if needed) back up the function stack. The calling of callback is how async knows your asynchronous function is done.