Better way to "loop" promises

Question

This is a post that might come across as quite conceptual, since I first start with a lot of pseudo code. - At the end you'll see the use case for this problem, though a solution would be a "tool I can add to my tool-belt of useful programming techniques".

The problem

Sometimes one might need to create multiple promises, and either do something after all promises have ended. Or one might create multiple promises, based on the results of the previous promises. The analogy can be made to creating an array of values instead of a single value.
There are two basic cases to be considered, where the number of promises is indepedented of the result of said promises, and the case where it is depedent. Simple pseudo code of what "could" be done.

for (let i=0; i<10; i++) {
    promise(...)
        .then(...)
        .catch(...);
}.then(new function(result) {
    //All promises finished execute this code now.
})

The basically creates n (10) promises, and the final code would be executed after all promises are done. Of course the syntax isn't working in javascript, but it shows the idea. This problem is relativelly easy, and could be called completely asynchronous.

Now the second problem is like:

while (continueFn()) {
    promise(...)
        .then(.. potentially changing outcome of continueFn ..)
        .catch(.. potentially changing outcome of continueFn ..)
}.then(new function(result) {
    //All promises finished execute this code now.
})

This is much more complex, as one can't just start all promises and then wait for them to finish: in the end you'll have to go "promise-by-promise". This second case is what I wish to figure out (if one can do the second case you can also do the first).

The (bad) solution

I do have a working "solution". This is not a good solution as can probably quickly be seen, after the code I'll talk about why I dislike this method. Basically instead of looping it uses recursion - so the "promise" (or a wrapper around a promise which is a promise) calls itself when it's fulfilled, in code:

function promiseFunction(state_obj) {
    return new Promise((resolve, reject) => {
        //initialize fields here
        let InnerFn = (stateObj) => {
            if (!stateObj.checkContinue()) {
                return resolve(state_obj);
            }
            ActualPromise(...)
                .then(new function(result) {
                    newState = stateObj.cloneMe(); //we'll have to clone to prevent asynchronous write problems
                    newState.changeStateBasedOnResult(result);
                    return InnerFn(newState);
                })
                .catch(new function(err) {
                    return reject(err); //forward error handling (must be done manually?)
                });
        }
        InnerFn(initialState); //kickstart
    });
}

Important to note is that the stateObj should not change during its lifetime, but it can be really easy. In my real problem (which I'll explain at the end) the stateObj was simply a counter (number), and the if (!stateObj.checkContinue()) was simply if (counter < maxNumber).

Now this solution is really bad; It is ugly, complicated, error prone and finally impossible to scale.
Ugly because the actual business logic is buried in a mess of code. It doesn't show "on the can" that is actually simply doing what the while loop above does.
Complicated because the flow of execution is impossible to follow. First of all recursive code is never "easy" to follow, but more importantly you also have to keep in mind thread safety with the state-object. (Which might also have a reference to another object to, say, store a list of results for later processing).
It's error prone since there is more redundancy than strictly necessary; You'll have to explicitly forward the rejection. Debugging tools such as a stack trace also quickly become really hard to look through.
The scalability is also a problem at some points: this is a recursive function, so at one point it will create a stackoverflow/encounter maximum recursive depth. Normally one could either optimize by tail recursion or, more common, create a virtual stack (on the heap) and transform the function to a loop using the manual stack. In this case, however, one can't change the recursive calls to a loop-with-manual-stack; simply because of how promise syntax works.

The alternative (bad) solution

A colleague suggested an alternative approach to this problem, something that initially looked much less problematic, but I discarded ultimatelly since it was against everything promises are meant to do.

What he suggested was basically looping over the promises as per above. But instead of letting the loop continue there would be a variable "finished" and an inner loop that constantly checks for this variable; so in code it would be like:

function promiseFunction(state_obj) {
    return new Promise((resolve, reject) => {
        while (stateObj.checkContinue())  {
            let finished = false;
            let err = false;
            let res = null;
            actualPromise(...)
                .then(new function(result) {
                    res = result;
                    finished = true;
                })
                .catch(new function(err) {
                    res = err;
                    err = true;
                    finished = true;
                });
            while(!finished) {
                sleep(100); //to not burn our cpu
            }
            if (err) {
                return reject(err);
            }
            stateObj.changeStateBasedOnResult(result);
        }
    });
}

While this is less complicated, since it's now easy to follow the flow of execution. This has problems of its own: not for the least that it's unclear when this function will end; and it's really bad for performance.

Conclusion

Well this isn't much to conclude yet, I'd really like something as simple as in the first pseudo code above. Maybe another way of looking at things so that one doesn't have the trouble of deeply recursive functions.

So how would you rewrite a promise that is part of a loop?

The real problem used as motivation

Now this problem has roots in a real thing I had to create. While this problem is now solved (by applying the recursive method above), it might be interesting to know what spawned this; The real question however isn't about this specific case, but rather on how to do this in general with any promise.

In a sails app I had to check a database, which had orders with order-ids. I had to find the first N "non existing order-ids". My solution was to get the "first" M products from the database, find the missing numbers within it. Then if the number of missing numbers was less than N get the next batch of M products.

Now to get an item from a database, one uses a promise (or callback), thus the code won't wait for the database data to return. - So I'm basically at the "second problem:"

function GenerateEmptySpots(maxNum) {
    return new Promise((resolve, reject) => {
        //initialize fields
        let InnerFn = (counter, r) => {
            if (r > 0) {
                return resolve(true);
            }
            let query = {
                orderNr: {'>=': counter, '<': (counter + maxNum)}
            };
            Order.find({
                where: query,
                sort: 'orderNr ASC'})
                .then(new function(result) {
                    n = findNumberOfMissingSpotsAndStoreThemInThis();
                    return InnerFn(newState, r - n);
                }.bind(this))
                .catch(new function(err) {
                    return reject(err); 
                });
        }
        InnerFn(maxNum); //kickstart
    });
}

---

EDIT: Small post scriptus: the sleep function in the alternative is just from another library which provided a non-blocking-sleep. (not that it matters).
Also, should've indicated I'm using es2015.

Answer 1

The alternative (bad) solution

…doesn't actually work, as there is no sleep function in JavaScript. (If you have a runtime library which provides a non-blocking-sleep, you could just have used a while loop and non-blocking-wait for the promise inside it using the same style).

The bad solution is ugly, complicated, error prone and finally impossible to scale.

Nope. The recursive approach is indeed the proper way to do this.

Ugly because the actual business logic is buried in a mess of code. And error-prone as you'll have to explicitly forward the rejection.

This is just caused by the Promise constructor antipattern! Avoid it.

Complicated because the flow of execution is impossible to follow. Recursive code is never "easy" to follow

I'll challenge that statement. You just have to get accustomed to it.

You also have to keep in mind thread safety with the state-object.

No. There is no multi-threading and shared memory access in JavaScript, if you worry about concurrency where other stuff affects your state object while the loop runs that will a problem with any approach.

The scalability is also a problem at some points: this is a recursive function, so at one point it will create a stackoverflow

No. It's asynchronous! The callback will run on a new stack, it's not actually called recursively during the function call and does not carry those stack frames around. The asynchronous event loop already provides the trampoline to make this tail-recursive.

The good solution

function promiseFunction(state) {
    const initialState = state.cloneMe(); // clone once for this run
    // initialize fields here
    return (function recurse(localState) {
        if (!localState.checkContinue())
            return Promise.resolve(localState);
        else
            return actualPromise(…).then(result =>
                recurse(localState.changeStateBasedOnResult(result))
            );
    }(initialState)); // kickstart
}

The modern solution

You know, async/await is available in every environment that implemented ES6, as all of them also implemented ES8 now!

async function promiseFunction(state) {
    const localState = state.cloneMe(); // clone once for this run
    // initialize fields here
    while (!localState.checkContinue()) {
        const result = await actualPromise(…);
        localState = localState.changeStateBasedOnResult(result);
    }
    return localState;
}

Answer 2

Let’s begin with the simple case: You have N promises that all do some work, and you want to do something when all the promises have finished. There’s actually a built-in way to do exactly that: Promise.all. With that, the code will look like this:

let promises = [];
for (let i=0; i<10; i++) {
    promises.push(doSomethingAsynchronously());
}

Promise.all(promises).then(arrayOfResults => {
    // all promises finished
});

Now, the second call is a situation you encounter all the time when you want to continue doing something asynchronously depending on the previous asynchronous result. A common example (that’s a bit less abstract) would be to simply fetch pages until you hit the end.

With modern JavaScript, there’s luckily a way to write this in a really readable way: Using asynchronous functions and await:

async function readFromAllPages() {
    let shouldContinue = true;
    let pageId = 0;
    let items = [];

    while (shouldContinue) {
        // fetch the next page
        let result = await fetchSinglePage(pageId);

        // store items
        items.push.apply(items, result.items);

        // evaluate whether we want to continue
        if (!result.items.length) {
            shouldContinue = false;
        }
        pageId++;
    }

    return items;
}

readFromAllPages().then(allItems => {
    // items have been read from all pages
});

Without async/await, this will look a bit more complicated, since you need to manage all this yourself. But unless you try to make it super generic, it shouldn’t look that bad. For example, the paging one could look like this:

function readFromAllPages() {
    let items = [];
    function readNextPage(pageId) {
        return fetchSinglePage(pageId).then(result => {
            items.push.apply(items, result.items);

            if (!result.items.length) {
                return Promise.resolve(null);
            }
            return readNextPage(pageId + 1);
        });
    }
    return readNextPage(0).then(() => items);
}

---

First of all recursive code is never "easy" to follow

I think the code is fine to read. As I’ve said: Unless you try to make it super generic, you can really keep it simple. And naming also helps a lot.

but more importantly you also have to keep in mind thread safety with the state-object

No, JavaScript is single-threaded. You doing things asynchronously but that does not necessarily mean that things are happening at the same time. JavaScript uses an event loop to work off asynchronous processes, where only one code block runs at a single time.

The scalability is also a problem at some points: this is a recursive function, so at one point it will create a stackoverflow/encounter maximum recursive depth.

Also no. This is recursive in the sense that the function references itself. But it will not call itself directly. Instead it will register itself as a callback when an asynchronous process finishes. So the current execution of the function will finish first, then at some point the asynchronous process finishes, and then the callback will eventually run. These are (at least) three separate steps from the event loop, which all run independently from another, so you do no have a problem with recursion depth here.

Answer 3

Sorry, this doesn't use Promises, but sometimes abstractions just get in the way.

This example, which builds from @poke's answer, is short and easy to comprehend.

function readFromAllPages(done=function(){}, pageId=0, res=[]) {
    fetchSinglePage(pageId, res => {
        if (res.items.length) {
            readFromAllPages(done, ++pageId, items.concat(res.items));
        } else {
            done(items);
        }
    });
}

readFromAllPages(allItems => {
    // items have been read from all pages
});

This has only a single depth of nested functions. In general, you can solve the nested callback problem without resorting to a subsystem that manages things for you.

---

If we drop the parameter defaults and change the arrow functions, we get code that runs in legacy ES3 browsers.

Answer 4

The crux of the matter seems to be that "the actual business logic is buried in a mess of code".

Yes it is ... in both solutions.

Things can be separated out by :

• having an asyncRecursor function that simply knows how to (asynchronously) recurse.
• allowing the recursor's caller(s) to specify the business logic (the terminal test to apply, and the work to be performed).

It is also better to allow caller(s) to be responsible for cloning the original object rather than resolver() assuming cloning always to be necessary. The caller really needs to be in charge in this regard.

function asyncRecursor(subject, testFn, workFn) {
    // asyncRecursor orchestrates the recursion
    if(testFn(subject)) {
        return Promise.resolve(workFn(subject)).then(result => asyncRecursor(result, testFn, workFn));
        // the `Promise.resolve()` wrapper safeguards against workFn() not being thenable.
    } else {
        return Promise.resolve(subject);
        // the `Promise.resolve()` wrapper safeguards against `testFn(subject)` failing at the first call of asyncRecursor().
    }
}

Now you can write your caller as follows :

// example caller
function someBusinessOrientedCallerFn(state_obj) { 
    // ... preamble ...
    return asyncRecursor(
        state_obj, // or state_obj.cloneMe() if necessary
        (obj) => obj.checkContinue(), // testFn
        (obj) => somethingAsync(...).then((result) => { // workFn
            obj.changeStateBasedOnResult(result);
            return obj; // return `obj` or anything you like providing it makes a valid parameter to be passed to `testFn()` and `workFn()` at next recursion.
        });
    );
}

You could theoretically incorporate your terminal test inside the workFn but keeping them separate will help enforce the discipline, in writers of the business-logic, to remember to include a test. Otherwise they will consider it optional and sure as you like, they will leave it out!