-1

I have an array of Promises that should be run in parallel and be finished as soon as possible. We have a Promise.all(arrayOfPromises) for that. But I'd like to have a kind of a modified Promise.all that can work with a dynamic array. Let me expalain what I mean.

I start with an array of 215 URLs (urlsArray) that I should send a request to. But I can process only 100 requests at a time. So I take the first 100 URLs and create a fetchArray array of those 100 fetch'es and then pass it to Promise.all(fetchArray). Now let's say one of the Promises resolved. I'd like to remove it from the fetchArray and then put the 101st fetch from urlsArray into fetchArray. I could use Promise.race() for this. But it would interrupt all other requests so I would lose all the progess. And it's not what I want since I have to get responses form all 215 URLs as soon as possible.

Also while processing the current requests, new URLs could be added to the urlsArray array. Potentially the urlsArray could be endless and new URLs could be added to the array again and again. So fetchArray should replace the just resolved Promise with the next one from urlsArray (if any) over and over again without losing the progress of other requests.

Is it even possible to implement this? Could you please give me any ideas, hints or code? Maybe some articles or tutorials about how to implement this?

forty5
  • 63
  • 4
  • 1
    *"Promises that should be run in parallel"*: Promises don't run. They are objects, but not functions. – trincot Jan 12 '23 at 19:07
  • Is an [Async task manager with maximum number of concurrent "running" tasks](https://stackoverflow.com/questions/54901078/async-task-manager-with-maximum-number-of-concurrent-running-tasks) what you are looking for? – trincot Jan 12 '23 at 19:09
  • 1
    I know. I say that just to make it sound simplier. "Promises should be pending in parallel" sounds a bit ugly although technically correct – forty5 Jan 12 '23 at 19:10

1 Answers1

1

A code like that should work:

(async() => {
  const promises = []
  // add 10 promises that will resolve between 1 and 4 seconds
  for (let i = 0; i < 10; i += 1) {
    addPromise(wait(Math.random() * 3000 + 1000))
  }
  
  while (promises.length) {
    // wait for promise to resolve
    const result = await Promise.race(promises.map(p => p.promise))
    //remove resolved promise
    promises.splice(promises.findIndex(p => p.id === result.id), 1)
    console.log(result.val)
    // add new promise
    addPromise(wait(Math.random() * 3000 + 1000))
    console.log('promises left', promises.length)
  }

  function addPromise(promise) {
    const id = crypto.randomUUID()
    const newPromise = promise.then(val => ({
      id,
      val
    }))
    
    promises.push({ id, promise: newPromise })
  }
})()

// create a promise that resolves after some time
function wait(time) {
  return new Promise(r => setTimeout(() => r(time), time))
}

Although, I would rather use rxjs for this

Konrad
  • 21,590
  • 4
  • 28
  • 64