2

I'm sending around 0 - 10 000 fetch requests with JavaScript.

Currently I'm returning a promise that contains a for loop that loops through the fetch requests. This is extremely slow however.

await new Promise((resolve, reject) => {
  for (let i = 0; i < 10 000; i++) {
    let promise = fetch(`website.com/page={i}`)
    .then() // process data

    promise_array.push(prom);


  Promise.allSettled(promise_array).then(() => {
    resolve();
  })
}

What is a more efficient way to do this?

M. Boyes
  • 41
  • 1
  • 5
  • 1
    10K requests all at once is probably too much for any web browser. Maybe you should try doing them in batches instead, maybe about 100 each batch? – yms Apr 03 '21 at 01:56
  • 2
    You are waiting for all promises to be resolved. This is going to be slow. Do you need all 10000 records at once? Otherwise, you could process each response as they come instead of waiting to accumulate them all. – elclanrs Apr 03 '21 at 01:57
  • 1
    If you do that you could get a too many request error from the API, also, do you need to handle the response of the promise ? – Juan Rambal Apr 03 '21 at 02:00
  • 1
    How many of those even get started? Why not shoot for some modest number of fetches to be in flight at any given time, and start new ones when earlier ones complete. Probably using only 6 underlying sockets anyway. You likely have have a saturated resource after you get the first dozen or so underway anyway. (Client/Server CPU or Network) – Wyck Apr 03 '21 at 02:03
  • Can any of that be cached? – charlietfl Apr 03 '21 at 02:07
  • 3
    Probably use a task queue with a limit to the maximum number of requests active at once. Your code doesn't seem like it'd work out of the box, although I get the gist of it. I recommend fixing it up. 10k requests is a lot and rather unusual -- more context might be needed to avoid an [XY problem](https://meta.stackexchange.com/questions/66377/what-is-the-xy-problem). – ggorlen Apr 03 '21 at 02:14
  • Avoid the [`Promise` constructor antipattern](https://stackoverflow.com/q/23803743/1048572?What-is-the-promise-construction-antipattern-and-how-to-avoid-it)! – Bergi Oct 22 '22 at 17:51

1 Answers1

-2

Use Promise.all(), like so: Promise.all([promise1, promise2, promise3, promise4])

This'll return the resolved responses

Michael Essiet
  • 37
  • 2