Run multiple promises simultaneously, but resolve in sequence (JS)

Question

Update 2023/08/03, 10:44: Modified my code to make clear the for loop could actually run for hours

I have an use case where I'd like to run multiple promises simultaneously, but resolve every promise in the order it was pushed into the queue. I came up with the solution below, but I'm not very happy with the approach. Any suggestions for improvements?

class PromiseQueue {
    #queue = [];

    enqueue(promise) {
        return new Promise((resolve) => {
            this.#queue.push({
                promise: promise(),
                resolve
            });
            
            if (this.#queue.length == 1) this.#dequeue();
        });
    }
    
    #dequeue() {
        let item = this.#queue[0];
        if (!item) return;
        item.promise.then((val) => {
            item.resolve(val);
            this.#queue.shift();
            this.#dequeue();
        });
    }
}

Usage

function sleep(ms) {
    return new Promise(resolve => setTimeout(resolve, ms));
}

function randomNumber(min, max) {
    return Math.floor(Math.random() * (max - min + 1)) + min;
}

async function* getSwaps() { 
// ....
}

async function run() {
    let queue = new PromiseQueue();
    let i = 0;
    // could run for hours
    for await (let logs of getSwaps() {
        let fn = async () => {
            await sleep(randomNumber(300, 1000));
            return i;
        }
        
        queue.enqueue(fn).then((val) => {
            console.log(val)
            // write data to database as soon as available
        });
        i++;
    }    
}

run();

Does this answer your question? [Promise.all: Order of resolved values](https://stackoverflow.com/questions/28066429/promise-all-order-of-resolved-values) — Justinas, Aug 03 '23 at 09:31
Simply push all promises into an array and await each of them in a loop. Your `PromiseQueue` complicated the process and did nothing more than that. According to [this post](https://stackoverflow.com/questions/35177230/are-promises-lazily-evaluated), a promise starts evaluating once it is constructed. So if you want them to run simultaneously, there is nothing to care about. Simply create all of them and await as you like. — Ricky Mo, Aug 03 '23 at 09:57

Ebod Shojaei · Answer 1 · 2023-08-03T10:30:36.233

0

You can instead use 'Promise.all()' which will execute your promises concurrently, returning them sequentially. Doing so will allow you to remove the class 'PromiseQueue' entirely, simplifying your code.

Make sure to revise your code by declaring 'queue' as an array, which will allow you to push the promises you already created in your 'fn' function. I've revised your code below:

function sleep(ms) {
    return new Promise(resolve => setTimeout(resolve, ms));
}

function randomNumber(min, max) {
    return Math.floor(Math.random() * (max - min + 1)) + min;
}

async function run() {
    let queue = [];
    for (let i = 0; i < 20; i++) {
        let fn = async () => {
            await sleep(randomNumber(300, 1000));
            return i;
        }
        queue.push(fn());
    }

    Promise.all(queue)
    .then((result) => {
        result.forEach((value) => {
            console.log(value);
        })
    })
}

run();

To keep the result in line with your previous code, please note that I included a loop that the Promise.all() will resolve, returning your values as individual elements instead of the complete array.

For more information about Promise.all(), I've included a snippet from MDN Web Docs:

The Promise.all() static method takes an iterable of promises as input and returns a single Promise. This returned promise fulfills when all of the input's promises fulfill (including when an empty iterable is passed), with an array of the fulfillment values. It rejects when any of the input's promises rejects, with this first rejection reason.

Hope this helps!

edited Aug 03 '23 at 10:30

answered Aug 03 '23 at 10:27

Ebod Shojaei

21
4

Thanks, but maybe my sample was simplified to much. Actually I call a generator function in the for loop that processes a bunch of data (takes hours to complete). But as soon as a promise returns in order I want to write the data to my database. This approach wouldn't work for me. – MrCoinSiX Aug 03 '23 at 10:41
I just modified my example to make it more clear. Sorry for the confusion – MrCoinSiX Aug 03 '23 at 10:50
Would it be possible to call your database function in the line following the console.log() statement in the Promise.all() function? Without knowing how you've implemented your code to write the data to the database, I'm not sure if my suggestion will work. – Ebod Shojaei Aug 03 '23 at 11:23
Would mean the database would be filled after the for loop, that runs for hours. Hence if the app crashes all data would be lost. Not my preferred solution to be honest with you. – MrCoinSiX Aug 03 '23 at 11:57
Is it possible to handle data processing/upload within an async function? That way, you can prefix with 'await' when you call your function inside the loop via 'Promise.all()' to synchronously run the function for each iterated data value. The sequential order should be preserved using "Promise.all()". You can include a '.catch()' to handle any errors from processing/writing to the database; there shouldn't be data loss if you define 'queue' outside the 'run()' function instead because the array should contain your original raw data. – Ebod Shojaei Aug 03 '23 at 12:04
Can you please give me an example in code? – MrCoinSiX Aug 03 '23 at 14:54
I think the original question has been answered. If possible, could you please open another question that describes the new problem regarding integrating the data functions you described? It would help to see a snippet of the code for those functions and also to know what database framework you’re using. I’ll try my best to help. Thanks! – Ebod Shojaei Aug 04 '23 at 16:39

Run multiple promises simultaneously, but resolve in sequence (JS)

1 Answers1