use async operation inside cycle for with timeout

Question

Problem: I have a list of users to delete. Not only the user himself is deleted, but also a significant amount of data associated with him. Some tables that are involved in deleting user information contain millions of records. All internal processes are started asynchronously when deleted. And if, when deleting one user, there are no problems, and all processes in the background are successful (that is, the client does not expect a response), then if you run several of them, this takes all the server resources, which naturally negatively affects the work of the site as long as it goes removal.

for (let i = 0; i < users.length; i++) {
    const user = users[i];
    await this.removeInstance(user, removedBy);
}

this.removeInstance contains many other asynchronous operations.

What I want: start an asynchronous operation this.removeInstance at 3 minute intervals. The speed of data deletion is not important to me, but according to my observations 1-2 minutes is exactly enough to "completely finish" with one user.

Note: this.removeInstance return removed user, but I cannot use this, since a dozen processes are already running in the background to clear his information.

Thanks for any help!

Answer 1

Sorry to bother you, but a solution has been found. How do I add a delay in a JavaScript loop?

Final:

public async removeBatchUsers(usersEmailsList: string[], removedBy: IIdentity) {
    const users: UserSchema[] = await User.find({userEmail: {$in: usersEmailsList}});

    const timer = ms => new Promise(res => setTimeout(res, ms));

    async function load (that) {
        for (let i = 0; i < users.length; i++) {
            const user = users[i];
            await that.removeInstance(user, removedBy);
            await timer(10000);
        }
    }

    load(this);
}

The solution above using Promise.allSettled is the most correct and safest. Tested it, in my case I had to additionally install the "promise.allsettled" library (23Kb)

Answer 2

Hi you can fake a sleep function and use it

function sleep(time = 0){
  return new Promise(resolve=>{
    setTimeout(()=>{
      resolve()
    },time)
  })
}

for (let i = 0; i < users.length; i++) {
    const user = users[i];
    await this.removeInstance(user, removedBy);
    await sleep(1000*60*3)
}

Answer 3

You can create a wrapper function that delays an async task completion to a minimum:

const wait = ms =>
  new Promise(resolve => setTimeout(resolve, ms));

const delayCompletion = minTime => task =>
  Promise.allSettled([task(), wait(minTime)])
    .then(([result]) => result.status === "fulfilled"  
      ? result.value 
      : Promise.reject(result.reason)
    );

Example:

const wait = ms =>
  new Promise(resolve => setTimeout(resolve, ms));

const delayCompletion = minTime => task =>
  Promise.allSettled([task(), wait(minTime)])
    .then(([result]) => result.status === "fulfilled"  
      ? result.value 
      : Promise.reject(result.reason)
    );
      

//showcase:

async function main() {
  console.log("-- start loop --");
  for(let i = 0; i < 3; i++) {
    console.log(`start iteration ${i}`);
    
    const delayAtLeastTwoSeconds = delayCompletion(2000);
    const oneSecondtask = () => wait(1000);
    
    await delayAtLeastTwoSeconds(oneSecondtask);
    
    console.log(`finish iteration ${i}`);
  }
  console.log("-- finish loop --");
}

main();

Promise.allSettled() waits until all promises are no longer pending. We give it one asynchronous task an it adds another for a minimum delay, so if task finishes early, then it still has to wait until the delay timer is finished. Conversely, if the delay timer finishes first but task has not, then that will still be awaited.

Promise.allSettled resolves to an array of promises where each has a status "fulfilled" or "rejected". We are only interested in the first item of the array because it's task - the second one is the delay. We just return its value if it was successful or reject with the original rejection reason, if it wasn't. That way delayCompletion still preserves the same semantics as the original promise and any code that consumed the original promise can transparently consume the promise from delayCompletion.

In some respects, this is the opposite of Promise.race() which instead resolves to the promise that resolves first. Instead we wait for the last one but we do have a fixed return value.

With this helper function, your code can be transformed like this:

const safeDelay = delayCompletion(3 * 60 * 1000); //3 minutes

for (let i = 0; i < users.length; i++) {
    const user = users[i];
    await safeDelay(() => this.removeInstance(user, removedBy));
}

Answer 4

If you're not too worried about about how long the whole process takes you can use a for..of loop with await to run this.removeInstance in series over the array:

for (let user of users) {
  await this.removeInstance(user)
}

This will wait for each user to be removed before starting the next - you could then return a http 202 to the client to indicate the message is received and is being processed, perhaps also implementing a polling endpoint to check in on the status of the work.