How to transform synchronous style code to asynchronous code?

Question

I have some synchronous code that looks like this:

function bulk_upload(files, callback) { 
  for (file in files) {
   sync_upload(file); // blocks till file uploads
  }
  print('Done uploading files');
  callback();
}

I now have to use an asynchronous API async_upload(file, callback) instead of sync_upload(file) to do the same. I have various options but not sure what is the best:

1) Use a sleep after the for-loop - that's a hack as I have to hope my timing is correct

2) Recursively chain my array:

function bulk_upload(files, callback) {
  if (files.length == 0) {
    print('Done uploading files');
    callback();
  } else {
    async_upload(files.removeLast(), function() { bulk_upload(files, callback); });
  }
}

This is not only hacky but sub-optimal as I could have uploaded my files in parallel using the new async_upload API but I ended up uploading sequentially.

3) Use a global counter:

function bulk_upload(files, callback) {
  uploads = 0
  for(file in files) {
    async_upload(file, function() { uploads++; }); 
  }
  while(uploads < files.length) ; // Empty spin - this is stupid
  print('Done uploading files');
  callback();
}

4) Slightly better counter (but still awful):

function bulk_upload(files, callback) {
  uploads = 0
  for(file in files) {
    async_upload(file, function() { 
      if (++uploads == files.length) { // this becomes uglier as I want to await on more stuff
        print('Done uploading files');
        callback();
      }; 
    }); 
  }
}

You can use futures to get this to work. There's a nice example [here][1]. [1]: http://stackoverflow.com/questions/5989538/nodejs-parallel-callback-design-pattern#answer-5994989 — Michael Anderson, Oct 01 '12 at 00:27
The futures answer is the cleanest way, and your 4) is the other mroe standard way, though often wrapped in a helper. I just wanted to point out that your 1) and 3) won't work anyway. Your spinlock will never finish because Javascript uses a single thread, so while your while is looping, it is impossible for the upload callback to run. — loganfsmyth, Oct 01 '12 at 01:38
Note that option 3) isn't viable; blocking in a tight loop will prevent any of `async_upload`s callbacks from running so you'll loop infinitely. — ebohlman, Oct 01 '12 at 01:38

score 3 · Accepted Answer · answered Oct 01 '12 at 02:06

You can use the async module's forEach method to do this:

function bulk_upload(files, callback) {
    async.forEach(files, async_upload(file, callback), function (err) {
        if (err) {
            console.error('Failed: %s', err);
        } else {
            console.log('Done uploading files');
        }
        callback(err);
    });
}

score 2 · Answer 2 · edited Oct 01 '12 at 01:34

Further to my comment, it looks like this code will suffice using futures (untested).

function aync_upload_promise(file) {
    // create a promise.
    var promise = Futures.promise();
    async_upload( file, function(err, data) {
        if (err) {
            // break it
            promise.smash(err);
        } else {
            // fulfill it
            promise.fulfill(data);
        }
    });
    return promise;
}

var promises = [];
for(var i=0; i<files.length; ++i )
{
   promises.push( aync_upload_promise( files[i] ) );
}

Futures
   .join( promises )
   .when( function() {
      print('Done uploading files');
      callback();
   } )
   .fail( function(err) { print('Failed :(', err); } )

How to transform synchronous style code to asynchronous code?

2 Answers2