2

I am trying to create an array of promises so I can batch promises instead of executing them all together to avoid shutting down my server. I want to execute up to 20 promises at a time. I have written the following code:

let promises = [];
let create;
let update;
let i=0;
let users = users.json;

if (users && usersjson.users.length) {
  for (const user of users) {
    if (userexists) {
      update = new promise(async (resolve,reject) => {
        //Function to update user
      });
      promises.push(update);
      i++;
    } else {
      create = new promise (async (resolve,reject) => {
        //Function to create a new user
      });
      promises.push(update);
      i++;
    }
  if (promises.length >= 20 || i >= usersjson.users.length) {
    await Promise.all(promises)
    .then((response)=>{
    console.log(response);
    promises=[];
  }).catch((err)=> {
    console.log(err);
  })
}
}
}

However, I found that the promises are executed when I define them, instead of being pushed into the array and executed when I call Promise.all, and I can't figure out why.

I would also prefer to have the Promise.all function continue to run even if a single promise was rejected, if that's possible with the way my code is built. I have a catch inside each promise in case the code fails, is that the correct way to do that?

Nimrod Yanai
  • 777
  • 2
  • 8
  • 30
  • Promises are not something that triggers later. A promise just marks marks that there is already an operation running and a value pending. It serves as a notification mechanism for when the value is available. – VLAZ Mar 09 '21 at 09:02
  • So in order to do that, I need to change all the promises to functions and push those into the array? – Nimrod Yanai Mar 09 '21 at 09:03
  • Promises do neither execute when you "define" them nor when you call `Promise.all()` but they are executed, when they are taken from the execution stack. And if you don't want to reject `Promise.all()` to reject, when one promise rejects, you will have to catch the rejection of each promise. But of course you then will have to check the result array – derpirscher Mar 09 '21 at 09:10
  • I see it mentioned several times, so to clear confusion - promises are not "executed". An asynchronous task is executed *which returns a promise*. A promise is not the task itself, it's not "running" or anything like that. When you order a pizza and get a receipt, *it's not the receipt that makes your pizza*. Your pizza is being made elsewhere by other people - the receipt is just a confirmation that your order is taken. When your number gets called at the end of the pizza making process, that is the promise notification mechanism. You can `.then(pizza => goHomeWith(pizza)).then(eat)` – VLAZ Mar 09 '21 at 09:21

2 Answers2

2

As mentioned in comments, promise objects actually get in "pending" state right after you executed an asynchronous operation and are "fulfilled" as soon as operation finishes.

So in your code you actually create and run async operations for all users, not only 20 of them.

There are two solutions for you. The main one is to create functions which return a promises and run only 20 at one time.

const jobs = users.map(user => () => user.update(...)); // array of functions
while (jobs.length > 0) {
   await Promise.all(jobs.splice(0,20).map(job => job())); // take 20 and process
}

An alternative solution is to use a library like bluebird which has a lot of useful methods for working with promises. One you'd probably like is map() which support concurrency limit.

Your second question was regarding errors in Promise.all what results in a fail of the whole series. In order to prevent that you may add .catch() to each job and, for example, return a null there or any other value which then will help you to determine that some operations failed. Of course, this approach will also prevent Promise.all from interrupting.

const jobs = users.map(user => () => user.update().catch(e => null));
Artem Arkhipov
  • 7,025
  • 5
  • 30
  • 52
  • Thanks! I'm not sure I understand how the map function work. I read the documentation, but it looks differently than how you wrote it. – Nimrod Yanai Mar 09 '21 at 10:40
  • 1
    In my examples `map` is a simple `Array.map` which I use on users array in order to get new array of functions. Bluebirds map is a different thing and you may need to spent a bit more time with the bluebird docs in order to understand it. Good luck! – Artem Arkhipov Mar 09 '21 at 11:13
  • Thanks @Artem, but I'm still confused. What functions do I put in this new "jobs" array? Instead of pushing the functions to "promises", I push them to "jobs"? And why do I need to attach it to the users element? – Nimrod Yanai Mar 10 '21 at 08:59
  • It is just an example. Main idea is to create array of functions which return promises. In my example I assumy that `user` object has `update` method which return promise, this made example shorter and more readable. In your case it may look in many other ways – Artem Arkhipov Mar 10 '21 at 09:13
  • So in my example, instead of doing a loop "for (const user of usersjson.users)", I will use "usersjson.users.map()" and will then put the functions inside the map function? What happens if I need to add a function that also has a ".then" after it? – Nimrod Yanai Mar 10 '21 at 09:16
1

A function in Javascript executes directly. Difference with a Promise is that you can wait for it in a Promise-style way. When the function is called, so like you're doing in the for loop, it executes it right away. Later on, with the Promise.all, you're just saying: "Wait for the promises I've defined in my array to complete" (and error if one of them fails).

About your second question, there's a way to wait for all promises to complete, and not throwing an error (and thus stopping). That's with Promise.allSettled. As the name suggests, it waits for all promises to settle.

However, keep in mind that the response will be different than the Promise.all function. As stated in the docs, it'll return an array per promise if the function is 'fulfilled' or 'rejected'.

Bart Versluijs
  • 186
  • 1
  • 8