Delay between HTTP requests in a loop

Question

I am looping in an array of products then for each product, I retrieve data from an API (I don't have control on it). Once data is collected for all products, I save the data in a file (that will then be imported later on).

  let dataToStore = [];
  let requests = products.map((product) => {
    axios('https://localhost:4000/getProduct/' + product.id).then((data) => {
      dataToStore.push({ product: data.response });
    }, 1000);
  });
  Promise.all(requests)
    .then(() => {
      saveDataInFile(dataToStore, filename);
    })
    .catch(function (err) {
      console.error("Promise.all error", err);
    });

My concern here is sometimes, there can be 1000+ products, which means 1000 requests, and get sometimes address not found and I could potentially slow down the website.

I would like to set a timeout between request, but have no idea how to do so with Promises here.

I tried to wrap the axios request in a setTimeout but then saveDataInFile gets called before everything is done.

Just a thought, If the backend is your own why not just make it so you get all the products you need in one response instead of getting them 1 by 1? — crimson589, Jul 10 '20 at 07:10
Probably what you need to do is to run N requests at a time where N is a smallish number in the range 2-10 rather than trying to run all the requests at once. There is code and explanation for how to do that here: [Promise.all() consumes all my ram](https://stackoverflow.com/questions/46654265/promise-all-consumes-all-my-ram/46654592#46654592). — jfriend00, Jul 10 '20 at 07:16
@crimson589 I don't have control of the server, otherwise there would be much better solution indeed — Mahcih, Jul 10 '20 at 07:28
@Mahcih oh, well then i think you should just use a normal loop instead of map. so you can set a timeout before continuing the loop. — crimson589, Jul 10 '20 at 07:40

Delay between HTTP requests in a loop

0 Answers0