promises.push() runs instantly and does not wait for promises.all()

Question

I have a nodejs function processReviews(workflow) that when called, is supposed to push multiple promises to an array promises[], then run them all with promises.all() after the for loop.

function examplePromiseFunc(){
    return new Promise((resolve, reject) => {
        console.log("examplePromiseFunc() INSIDE ")
        resolve('done')
    })
}

async function processReviews(workflow){
        //get objects from s3
        let allObjects = await getAllObjects(workflow);
        allObjects = allObjects.filter(obj => obj.Key.includes('output.json'))
        console.log(`found ${allObjects.length} .json files.`)

        const promises = [];
        for (let i = 0; i < allObjects.length; i++) {
            console.log('i=',i,' pushing to promises[]')
            promises.push( examplePromiseFunc() )
        }

        const result = await Promise.all(promises)
        console.log('running, result = ', result);
}

But when I run my code, the output looks like this:

found 697 .json files.
i= 0  pushing to promises[]
examplePromiseFunc() INSIDE
i= 1  pushing to promises[]
examplePromiseFunc() INSIDE
i= 2  pushing to promises[]
examplePromiseFunc() INSIDE
i= 3  pushing to promises[]
examplePromiseFunc() INSIDE
...

Which means that each time I push a promise to my promises[] array (promises.push( await examplePromiseFunc() )) the function examplePromiseFunc() is getting called instantly and is not waiting.

I want my function to only get called when I run await Promise.all(promises) at the end, is there something I'm missing? are my async functions causing a problem? I've been reading up on javascript promises.all and this seems like a fine implementation.

Answer 1

The issue is that you are already using await inside your loop, which means the loop will "wait" and process the items sequentially.

Instead, you should just add the promises to the array, then await all of them at the end like you had:

async function processReviews(workflow) {
  //get objects from s3
  const allObjects = await getAllObjects(workflow);

  const promises = [];
  for (let i = 0; i < allObjects.length; i++) {
    // Don't await the promise here, just start it and add it to the array.
    promises.push(examplePromiseFunc(allObjects[i]));
  }
  const result = await Promise.all(promises)
  console.log(result);
        
}

Answer 2

There is a fundamental misunderstanding here of of how the Promise constructor works.

The constructor takes a single function argument known as the executor:

new Promise( executor)

The executor function is called synchronously during construction with two function arguments commonly called resolve and reject:

executor( resolve, reject)

The promise executor is responsible for initiating an asynchronous operation (usually) and make the resolve and reject functions available to code that handles operation completion and error handling, often within a call back function.

Hence the code

    for (let i = 0; i < allObjects.length; i++) {
        console.log('i=',i,' pushing to promises[]')
        promises.push( examplePromiseFunc() )
    }

calls examplePromiseFunct multiple times and within that function, the executor for the promise returned is called synchronously (by Promise) during construction. Hence the log is as one would expect: a log of "examplePromiseFunc() INSIDE" each time examplePromiseFunc is called.

This misunderstanding probably leads to the second one:

Promise.all does not "run" promises - promises are passive objects that respond in deterministic ways to calling their associated resolve and reject functions, by calling either fulfilled or rejected handlers attached to them - or chaining off another promise if resolved with a promise.

Promise.all simply returns a promise that is fulfilled with an array of the fulfilled results of its argument promises, or is rejected with the rejection reason or the first promise in its argument array that becomes rejected. It has an executor in native code that effectively attaches then handlers to promises in its argument array and then passively waits (i.e. it returns to the event loop) until argument promises are settled one at a time.

Answer 3

that's because promises work this way only. When you are pushing a promise into an array you are waiting for it already(inside the loop) i.e if you would not wait for them to execute then also they would execute, with or without await promise.all, Also it is possible that all the promises already resolved before you pass that array in promise.all. The below function would also resolve all promises without the promise all.

async function processReviews(workflow){
        //get objects from s3
        let allObjects = await getAllObjects(workflow);
        allObjects = allObjects.filter(obj => obj.Key.includes('output.json'))
        console.log(`found ${allObjects.length} .json files.`)

        const promises = [];
        for (let i = 0; i < allObjects.length; i++) {
            console.log('i=',i,' pushing to promises[]')
            promises.push( examplePromiseFunc() )
        }
}

Also, you should not use promise.all without limit as it might reach limitations of your hardware. Using a map with a limit can reduce your problem.

async function processReviews(workflow) {
  //get objects from s3
  let allObjects = await getAllObjects(workflow);
  allObjects = allObjects.filter(obj => obj.Key.includes("output.json"));
  console.log(`found ${allObjects.length} .json files.`);

  for (let i = 0; i < allObjects.length; i = i + PROMISE_LIMIT) {
    const objects = allObjects.slice(i, i + PROMISE_LIMIT);
    const result = await Promise.all(objects.map(elem => examplePromiseFunc()));
    console.log("running, result = ", result);
  }
}

Answer 4

Nothing just you have to use for in or for of loop in this type of case because forEach is some time not working in asynchronous function.