How to know when all Promises are Resolved in a dynamic "iterable" parameter?

Question

My problem is that I don't know how to know when a dynamic promise array has all the promises resolved.

Here an example:

var promiseArray = [];
promiseArray.push(new Promise(){/*blablabla*/});
promiseArray.push(new Promise(){/*blablabla*/});
Promise.all(promiseArray).then(function(){
    // This will be executen when those 2 promises are solved.
});
promiseArray.push(new Promise(){/*blablabla*/});

I have a problem here. The Promise.all behavior will be executed when the previous 2 promises are solved, BUT, before those 2 promises were solved, a third promise where added and this new one won't be take in account.

So, what I need, is say something like: "Hey Promise.all, you have a dynamic array to check". How can I do it?

Remember that this is just an example. I know I can move the line Promise.all to the last line, but actually the new promises are added dynamically when another promises are solved, and the new promises could add new promises as well, so, it's a really dynamic array.

The real use case that I have is something like this:

1. I use Twitter API to check if there are new Tweets (using the Search Api).
2. In case I found new Tweets, I add it to a MongoDB (here we have Promises).
3. In case that those new Tweets are related to a user that I do not have in my MongoDB (here we have new promises because I have to go to MongoDB to check if I have that user), we go to Twitter API to get user info (more promise) and we add those new users to MongoDB (yes, more promises).
4. Then, I go to MongoDB to insert new values to associate the new tweets with those new users (more promises! wiii!).
5. When all the queries to MongoDB are Resolved (all the selects, updates, inserts), close the MongoDB connection.

Another hard example:

var allPromises = [];

allPromises.push(new Promise(function(done, fail){
    mongoDB.connect(function(error){
        //Because mongoDB works with callbacks instead of promises
        if(error)
            fail();
        else
            ajax.get('/whatever').then(function(){
                if (somethingHappens) {
                    allPromises.push(new Promise(function(done, fail){ //This promise never will be take in account
                        // bla bla bla
                        if (somethingHappens) {
                            allPromises.push(new Promise(function(done, fail){ //This promise never will be take in account
                                // bla bla bla
                            }));
                        } else {
                            ajax.get('/whatever/2').then(function(){
                                if (somethingHappens) {
                                    allPromises.push(new Promise(function(done, fail){ //This promise never will be take in account
                                        // bla bla bla
                                    }));
                                }
                            });
                        }
                    }));
                } else {
                    ajax.get('/whatever/2').then(function(){
                        if (somethingHappens) {
                            allPromises.push(new Promise(function(done, fail){ //This promise never will be take in account
                                // bla bla bla
                                    if (somethingHappens) {
                                        allPromises.push(new Promise(function(done, fail){ //This promise never will be take in account
                                            // bla bla bla
                                        }));
                                    } else {
                                        ajax.get('/whatever/2').then(function(){
                                            if (somethingHappens) {
                                                allPromises.push(new Promise(function(done, fail){ //This promise never will be take in account
                                                    // bla bla bla
                                                }));
                                            }
                                        });
                                    }
                            }));
                        }
                    });
                }
            });
    });
}));

Promise.all(allPromises).then(function(){
    // Soooo, all work is done!
    mongodb.close()!
});

So, now, a beauty example. We need to call the showAllTheInformation function when the last (we don't know which is the last) promise is called. How do you do it?:

var name = 'anonimus';
var date = 'we do not know';

function userClikOnLogIn() {
    $http.get('/login/user/password').then(function(data){
        if (data.logguedOk) {
            $http.get('/checkIfIsAdmin').then(function(data){
                if (data.yesHeIsAnAdmin) {
                    $http.get('/getTheNameOfTheUser').then(function(data){
                        if(data.userHasName) {
                            $http.get('/getCurrentDate').then(function(data){
                                currentDate = data.theNewCurrentDate;
                            });
                        }
                    });
                }
            });
        }
    });
}

function showAllTheInformation() {
    alert('Hi ' + name + ' today is:' + date);
}

here another example with more context: https://jsfiddle.net/f0a1s79o/2/

Answer 1

You can make a neat little recursive function to wrap Promise.all to handle additions to the original promise:

/**
 * Returns a Promise that resolves to an array of inputs, like Promise.all.
 *
 * If additional unresolved promises are added to the passed-in iterable or
 * array, the returned Promise will additionally wait for those, as long as
 * they are added before the final promise in the iterable can resolve.
 */
function iterablePromise(iterable) {
  return Promise.all(iterable).then(function(resolvedIterable) {
    if (iterable.length != resolvedIterable.length) {
      // The list of promises or values changed. Return a new Promise.
      // The original promise won't resolve until the new one does.
      return iterablePromise(iterable);
    }
    // The list of promises or values stayed the same.
    // Return results immediately.
    return resolvedIterable;
  });
}

/* Test harness below */

function timeoutPromise(string, timeoutMs) {
  console.log("Promise created: " + string + " - " + timeoutMs + "ms");
  return new Promise(function(resolve, reject) {
    window.setTimeout(function() {
      console.log("Promise resolved: " + string + " - " + timeoutMs + "ms");
      resolve();
    }, timeoutMs);
  });
}

var list = [timeoutPromise('original', 1000)];
timeoutPromise('list adder', 200).then(function() {
  list.push(timeoutPromise('newly created promise', 2000));
});
iterablePromise(list).then(function() { console.log("All done!"); });

In ES6 with lambdas and without comments, this can be even shorter:

function iterablePromise(iterable) {
  return Promise.all(iterable).then((resolvedIterable) => {
    if (iterable.length != resolvedIterable.length) {
      return iterablePromise(iterable);
    }
    return resolvedIterable;
  });
}

Or, as Rads expressed with async/await in their answer, but as a function:

async function iterablePromise(iterable) {
  let resolvedIterable = [];
  while (iterable.length !== resolvedIterable.length) {
    resolvedIterable = await Promise.all(iterable);  // implicit "then"
  }
  return resolvedIterable;
}

Bear in mind that this only covers addition, and that it's still a little dangerous: You need to ensure that the callback order is such that any promises in flight add themselves to the list before the Promises.all callback can be invoked.

Answer 2

I know I am late to the party here. However, for those who felt sad and didn't find a quick answer, here is a dirty (explanation later) way of moving ahead without having to re-architect IF you are sure that a NEW promise will NOT be added AFTER completion of all existing promises.

var promiseArray = [], completedPromises = [];
promiseArray.push(new Promise(){/*blablabla1*/});
promiseArray.push(new Promise(){/*blablabla2*/});
while(completedPromises.length != promiseArray.length) completedPromises = await Promise.all(promiseArray);

Elsewhere in the code (before completion of all previous promises:

promiseArray.push(new Promise(){/*blablabla3*/});

Hope this helps someone. I created a stack overflow account after years of freeloading only for this :)

Answer 3

There's no way out. You have to put all the promises in the array before calling Promise.all in it. In the example you presented, that's as simple as moving the last line to the top.

In case you are asynchronously filling the array, you should get a promise for that array, and use .then(Promise.all.bind(Promise)). If you don't know when you stop adding promises, this is impossible anyway as they might never all be resolved at all.

---

Regarding your "beauty example", you will want to learn about the magic of chaining. As I previosly said in the comments, you have to return a promise from every function in which you are doing anything asynchronous. Indeed, just add the missing returns:

function userClikOnLogIn() {
    return $http.get('/login/user/password').then(function(data){
//  ^^^^^^
        if (data.logguedOk) {
            return $http.get('/checkIfIsAdmin').then(function(data){
//          ^^^^^^
                if (data.yesHeIsAnAdmin) {
                    return $http.get('/getTheNameOfTheUser').then(function(data){
//                  ^^^^^^
                        if(data.userHasName) {
                            return $http.get('/getCurrentDate').then(function(data){
//                          ^^^^^^
                                currentDate = data.theNewCurrentDate;
                            });
                        }
                    });
                }
            });
        }
    });
}

userClikOnLogIn().then(function showAllTheInformation() {
//               ^^^^^ now you can chain onto it!
    alert('Hi ' + name + ' today is:' + date);
});

There is no array of promises here that dynamically grows, it's just that every function is returning a promise for the (asynchronous) result of the things it does.

Answer 4

If you can instrument the promises or their usage, and scoping issues allow for it, then I think you could approach the problem more simply: how many promises are oustanding?

In other words, you don't need to keep track of all the promises, just count them.

var outstanding = 0;

var p1 = new Promise(){/*blablabla*/};
var p2 = new Promise(){/*blablabla*/};

++outstanding;
p1.then( (data) => { ...
  if (0 >= --outstanding) 
    // All resolved!
}

// dynamic set of promises, so later we decide to add another:
var p3 = new Promise(){/*blablabla*/};
++outstanding;
p3.then( ... );  // as above

To improve the above, wrap it all into the meta-promise (equivelent to the one that would be returned by Promise.all for a static set of promises)...

  // Create a promise that tracks completion of a dynamic set of instrumented promises.
  getCompletionP() { 
    let rslv = null;
    const p = new Promise(
      function(resolve, reject) {
        rslv = resolve;
      } );
    p.resolve = rslv;
    assert( p.resolve );
    p.scheduled = 0;
    p.onSchedule = function() { ++this.scheduled; };
    p.onComplete = function()  { if (0 >= --this.scheduled) this.resolve(); };
    return p;
  }

Now call cp.onSchedule() before every call to then(), and cp.onComplete at the end of each then(), and cp will resolve after all your promises then functions are complete. (You would need to handle the catch statements too, of course.)

This will resolve when all code scheduled via Promise.then calls is complete, whereas the question asks for something that will resolve when all promises are resolved. That could be achieved by adding calls after the resolve statement of the promises instead, but that is impossible if using 3rd party libraries, and I think they would be functionally the same.

This won't work for all cases, but since the accepted answer is that it (dynamic set of promises) can't be done, I figure this could still be useful though it got more complicated (messy) as I wrote it out!

Answer 5

@JeffBowman and @Bergi have the right idea: recursive waiting and counting promises. Here is my implementation in Coffeescript )

Promise = require 'bluebird'

class DynamicPromiseCollection

    promises = []

    add:(p)->
        promises.push p

    wait_for_all:->
        #
        # Wait for all current promises, then check for new promises...
        # ...if there are new promises, then keep waiting ( recursively ).
        #
        # Resolve only when all promises are done, and there are no new promises.
        #
        make_promise = ->
            num_before = promises.length
            p = Promise.all(promises).then ->
                num_after = promises.length
                if num_after > num_before
                    return make_promise() # recursive -- wait again
                else
                    return true # all done now
        p = make_promise()
        return p


#
# let's test this...
#
promises = new DynamicPromiseCollection()


#
# pretend to get remote data
#
get_remote_data = ->
    new Promise (resolve,reject)->
        setTimeout ->
            resolve "data"
        ,500

#
# get data, wait, then get more data...
#
promises.add get_remote_data().then (data)->
    console.log "got " + data
    promises.add get_remote_data().then (data)->
        console.log "got " + data

#
# this should wait for both data
#
promises.wait_for_all().then ->
    console.log "...and wait_for_all is done."