Run Plenty of promises sequentially

Question

A nodejs project.I've tried to run plenty(about 100k) task sequently with promises. What I can do is converting it to a workOnebyOne function with Q. Is there a better way to do this?

function workOnebyOne(items, worker) {
  var deferred = Q.defer()

  function _doNext() {
    if (items.length === 0) {
      deferred.resolve()
      return
    }
    var item = items[0]
    synchronize(worker, item)
      .then(function (result) {
        items = items.slice(1)
        deferred.notify({
          item: item,
          result: result
        })
        _doNext()
      }, function () {
        items = items.slice(1)
        _doNext()
      })
  }

  _doNext()

  return deferred.promise
}

utils.workOnebyOne(tasks, workerFunction)

Using an index instead of slicing the array is probably going to be more efficient. — Dark Falcon, Dec 05 '14 at 16:02
possible duplicate of [How to sequentially run promises with Q in Javascript?](http://stackoverflow.com/q/18386753/1048572)? — Bergi, Dec 13 '14 at 12:46
@Bergi just a simple helper to synchronize a async function into a sync function and pass parameter to the worker function — fxp, Dec 15 '14 at 06:05
@fxp: You mean a helper that calls an async function which expects a callback and returns a promise for that? You cannot "synchronize" an async function. — Bergi, Dec 15 '14 at 16:30
[Kindly check answer by lioned, it worked in my case. Hope it will help you also][1] [1]: https://stackoverflow.com/questions/27322994/nodejs-q-chaining-promises-sequentially — keshav, Aug 22 '15 at 13:55

score 2 · Answer 1 · answered Dec 05 '14 at 23:55

You are basically re-implementing queuing here. In Bluebird promises (Which are also much faster and consume a lot less memory which helps with 100K tasks) you'd use Promise.each.

In Q you can typically use .reduce on the tasks array to queue them at once - however with 100K elements creating a 100K promise queue in Q promises would crash node (again, this is Q, Bluebird or when promises) would handle it just fine). This (incorrect here) solution would look something like:

var res = tasks.reduce(function(p, c){
    return p.then(function(){ return workerFunction(c); });
}, Q());

For short queues ( < 500 promises in Q) this works nicely.

So because of the old library choice and because of the large number of promises involved you can't realistically solve it elegantly, using a callback queue like approach is pretty close to your only way. I'd also avoid notify as it is being removed (even from Q) and is generally a bad API (doesn't compose well).

Just curious, what exactly is crashing Q with long sequences of promises? Do you have a link on the issue at hand? — Bergi, Dec 13 '14 at 12:47
It's quite simple - the same thing would happen to Bluebird too (although at a much higher number of promises or even to a regular array of functions (at an even higher threshold) - it simply consumes too much memory. A Q promise takes significantly more memory than a Bluebird promise which uses bit flags for states. — Benjamin Gruenbaum, Dec 13 '14 at 12:50
Ah OK, I thought it would be limited by the call stack or something like that. — Bergi, Dec 14 '14 at 11:00

ciekawy · Answer 2 · 2015-03-10T15:08:30.787

1

I've spent some time finding simple and elegant solution. I have found only some hints and discussions but without ready example. Finally I have found discussion on https://github.com/kriskowal/q/issues/606 and as a result what have worked for me can be isolated and generalised like this:

function workOneByOne(items, someAsyncFuntionReturningPromise) {
  var lastResultPromise = items
    .map(function(item) {
      return function(previousResult) {
        /*
         * this function has to:
         * - process result from previous item processed if any
         * - call the async job
         * - return promise of the job done
         */
        if (previousResult) {
          // process result here
        }

        return someAsyncFuntionReturningPromise(item);
    }})
    .reduce(Q.when, Q());

    return lastResultPromise;
}

and in case there is no function returning promise available you can call above with

workOneByOne(items, Q.nfbind(someAsyncFunctionWithCallback))

edited Mar 10 '15 at 15:08

answered Mar 10 '15 at 11:37

ciekawy

2,238
22
35

Uh, that shouldn't be `someAsyncFunctionWithCallback`. Why not `someAsyncFuntionReturningPromises`? – Bergi Mar 10 '15 at 14:03
Sometimes it may be some external function not returning any promise. Eg my particular case was that I was originally using node http requests and they are not promises. Then I've switched to q-io so can just return the promise as you're suggesting. But just in case solution proposed above takes into account the cas of `someAsyncFunctionWithCallback` ;) – ciekawy Mar 10 '15 at 14:54
1

That case should be solved by [`Q.nfbind`](https://github.com/kriskowal/q/wiki/API-Reference#qnfbindnodefunc-args) rather. – Bergi Mar 10 '15 at 14:57
In the meantime basing on your commend I've updated the answer exactly with Q.nfbind :) Also I've found that item handling was missing. So now the answer has been corrected. – ciekawy Mar 10 '15 at 15:10

score 1 · Answer 3 · answered Dec 01 '18 at 22:31

Ultra simple solution if await / async is available.

// functions is an array of functions that return a promise.
async function runInSequence(functions) {
  const results = [];

  for (const fn of functions) {
    results.push(await fn());
  }

  return results;
}

And we can use it like this:

/**
 * Waits "delay" miliseconds before resolving the promise.
 * @param {Number} delay The time it takes to resolve the promise.
 * @param {any} value The resolving value.
 */
function promisedFunction(delay, value) {
  return new Promise(resolve => {
    setTimeout(() => resolve(value), delay);
  });
}

console.time("execution");
runInSequence([
    promisedFunction.bind(this, 1000, 1),
    promisedFunction.bind(this, 1000, 2),
    promisedFunction.bind(this, 1000, 3)
]).then(results => {
    console.timeEnd("execution");
    console.log(results);
});

Working RunKit example here.

Run Plenty of promises sequentially

3 Answers3

Linked