Recursive Promise.all?

Question

I'm trying to wait for a bunch of promises to complete. I know I can do this with Promise.all, but I can't figure out what to do when one of those promises pushes a new promise into the promise list.

Example:

asyncFunction(...).then(result => {
    // do something
});


asyncFunction(...).then(result => {
    for(let row of result) {
        asyncFunction(row);
    }
});


console.log(promises.length); // 2
await Promise.all(promises);
console.log(promises.length); // 5

Where asyncFunction is something like:

const asyncFunction = (...args) => {
    let result = new Promise((resolve, reject) => {
        pool.query(...args, (err, rows, fields) => {
            if (err) {
                reject(err);
            } else {
                resolve(rows); 
            }
        });
    });
    promises.push(result);
    return result;
};

What's happened here is that the first two calls to asyncFunction push promises into my promises array, so promises.length is 2. However, after waiting for them to complete, the 2nd one pushes a bunch of new promises into the array after Promise.all has already evaluated that array, so they're not awaited.

So, how can I wait for all the promises, plus any new ones? Simply calling await Promise.all(promises) twice would work in this example, but this could go on ad infinitum.

Don't push to the pool at arbitrary times. Wait until the pool is filled, `then` call `Promise.all` on it. Don't pool everything together in one array if you need to push at distinct times, rather call `Promise.all` multiple times. — Bergi, Jun 29 '16 at 18:12
@Bergi The new promises are created as a result of a previous async result. — user2864740, Jun 29 '16 at 18:15
Does [this](http://stackoverflow.com/q/37801654/1048572) help? — Bergi, Jun 29 '16 at 18:16
@user2864740: So? Then those new promises should be `Promise.all`'d, and be chained to the previous async result via `then` so that you get a promise for them. — Bergi, Jun 29 '16 at 18:17
@Bergi Yes, I think that's more or less the same question. The posted solution could probably be made slightly more efficient by not re-evaluating the already checked promises though. — mpen, Jun 29 '16 at 18:24
Possible duplicate of [How to know when all Promises are Resolved in a dynamic "iterable" parameter?](https://stackoverflow.com/questions/37801654/how-to-know-when-all-promises-are-resolved-in-a-dynamic-iterable-parameter) — Jeff Bowman, Jul 17 '19 at 02:15

TbWill4321 · Accepted Answer · 2016-06-29T19:01:36.317

5

You can write a recursive all function that re-evaluates the status of the list after each iteration:

function recursiveAll( array ) {
    // Wait for the promises to resolve
    return Promise.all( array ).then(function( result ) {
        // If no new promises were added, return the result
        if ( result.length == array.length )
            return result;
        // If new promises were added, re-evaluate the array.
        return recursiveAll( array );
    });
}

edited Jun 29 '16 at 19:01

answered Jun 29 '16 at 18:16

TbWill4321

8,626
3
27
25

2

You don't need to store `prevLength`, you can just use `result.length` – Bergi Jun 29 '16 at 18:19
Wouldn't `return recursiveAll(array.slice(result.length))` be more efficient? i.e. only check the new elements. – mpen Jun 29 '16 at 22:19
1

No, then we would have a new array with a new reference, and any items added to the original array would not be available in the next recursion. – TbWill4321 Jun 30 '16 at 17:24

score 2 · Answer 2 · answered Jun 30 '16 at 13:44

2

That's the problem with functions that have side effects.

Your code is a bit to abstract to give you a moe precise answer, but I would reccomend you to completely drop the promises-Array.

const asyncFunction = (...args) => {
    return new Promise((resolve, reject) => {
        pool.query(...args, (err, rows, fields) => {
            if (err) {
                reject(err);
            } else {
                resolve(rows); 
            }
        });
    });
}

and

asyncFunction(...).then(result => Promise.all(result.map(row => asyncFunction(row))) );

As far as I understand your intentions, this should result in a nested structure of Arrays within Arrays, in the worst case, wich now only has to be flattened to get ALL values, and those wich are triggered by the previous ones.

answered Jun 30 '16 at 13:44

Thomas

3,513
1
13
10

I tried doing it that way for a bit, it gets so messy. Its fine for a simple loop, but if you're doing a bunch of random stuff, then you have to collect up all these 'loose' promises so that you can return a `Promise.all` at the end. I just wanted something automatic; I don't care when all the individual promises complete, just when they're *all* done. – mpen Jun 30 '16 at 21:16
I agree, but due to the level of abstraction in your code-sample, I can't give you a better example/answer. Maybe you can show the concrete Problem, so we can discuss this on some real code. – Thomas Jul 01 '16 at 03:34
_That's the problem with functions that have side effects._ Good answer! – Jul 02 '16 at 21:18

score 0 · Answer 3 · answered Jun 29 '16 at 18:17

0

I think simply replacing Promise.all with this should work fine:

while(promises.length) {
    await promises.shift();
}

answered Jun 29 '16 at 18:17

mpen

272,448
266
850
1,236

1

`await` is not exaclty ES6. – Bergi Jun 29 '16 at 18:18
@Bergi No... it isn't, I'll retag the question. Although my original question already had an `await` in it. I guess an await-less version would be nice too. – mpen Jun 29 '16 at 18:19
1

The problem with this answer is that it will evaluate all promises in *sequence*, not in *parallel*. – TbWill4321 Jun 29 '16 at 19:02
@TbWill4321 What do you mean by "in parallel"? JS is single-threaded. The 3rd promise might resolve before the 1st and then the first iteration of this loop will hang longer than it needs to, *but* we're waiting for *all* the promises to complete anyway, so the *total* time remains unchanged, no? I guess the difference is if there's a rejection: `Promise.all` would bail sooner. – mpen Jun 29 '16 at 21:54
While JS is single-threaded, it does not synchronously wait for a Promise before moving to the next. A good example is multiple AJAX calls, which can be done in parallel. The point is starting as many async calls at the same time as possible, not waiting for the last before moving on to the next. – TbWill4321 Jun 30 '16 at 17:26
1

@TbWill4321 Indeed, but in my scenario, I'm firing them all off in parallel, I'm only awaiting them in sequence. – mpen Jun 30 '16 at 21:18

Recursive Promise.all?

3 Answers3