How to handle a lof of Ajax requests on a loop and prevent "Too many requests" using JavaScript

Question

I'm trying to save some information from a web service, it is a lot of data and I have to make a lot of Ajax requests, after many requests so quick, the server justs throws "Too many requests" which makes sense...

My code looks something like this:

function getDataFromWS()
{
    define some vars
    $ make an ajax request to MY database to get every item i will request on the external WS 
    (that query returns like 150 items to arrayFromResponse)
    
    //Loop into the array with 150 items
    arrayFromResponse.forEach((element)=>{
        //For loop to make request for each day of the month e.g. August 31 days
        for(let i = 1; i <= 31; i++){
            //change uri to match each day using i
            uriData.date = '2020-08-0'+i;

             //This is where after many requests it throws error 429 "Too many"
                $.ajax({
                    data: uriData,
                    url: uri,
                    type: 'GET',
                    async: false,
                    success : function(response){
                        
                            //Save data from response to some arrays I use
                    
                    },
                    error: function(response){
                        console.log(response);
                        console.log('Error');
                    }
                });
        } 
    })
    //After all that looping and requests make a CSV file
    buildCSVfile(dataFromResponses);
}

I need all this code to be synchronous, I assume that I need to use something like setTimeout or so because I tried it but then the buildCSVfile() function executes before the loops, I guess the delay should be after each ajax request in the date for loop. Maybe 10 seconds per request so it won't say too many requests? All the core code is in the success function. I don't need this to be fast, just to make sure to get all the info, per item and per day of the month.

I appreciate any help given.

"I need all this code to be synchronous" Why? BTW, if you add `setTimeout`, then it will no longer be synchronous. — Heretic Monkey, Oct 14 '20 at 15:54
@HereticMonkey Because of the CSV file with the saved data, I just need a delay for each ajax request and at the end of like 4500 requests, save the file with everything. It works right now but just for like 50 requests. — Hernán Casillas, Oct 14 '20 at 15:57
Consider a different technique where you don't need to do it synchronously. Look at answers to [Sending one AJAX request at a time from a loop](https://stackoverflow.com/q/5066002/215552) — Heretic Monkey, Oct 14 '20 at 16:21

Ashwin · Answer 1 · 2020-10-14T16:59:37.063

I suggest using jQuery Deffered since you already use jQuery.

Read up on jQuery Deferred to understand the inner workings. It basically implements a system based on Promises.

Your code could end up something like:

function getDataFromWS()
{
  var deferred = new $.Deferred();
  var dataFromResponses = [];
  //   define some vars
  //   $ make an ajax request to MY database to get every item i will request on the external WS 
  //   (that query returns like 150 items to arrayFromResponse)

  var uri = "https://jsonplaceholder.typicode.com/todos/";
  var arrayFromResponse = ["SOME DATA"];
  //Loop into the array with 150 items
  arrayFromResponse.forEach((element)=>{
    //For loop to make request for each day of the month e.g. August 31 days
    for(let i = 1; i <= 10; i++){
      var uriData = {};
      //change uri to match each day using i
      uriData.date = '2020-08-0'+i;

      var def = new $.Deferred();
      //This is where after many requests it throws error 429 "Too many"
      $.ajax({
        data: uriData,
        cache: false,
        url: uri + i,
        type: 'GET',
        async: false,
        success : function(response){
          console.log("Got data at day " + i);
          dataFromResponses.push(response);
        },
        error: function(response){
          deferred.reject(response);
          console.log('Error');
        }
      });
    }
  });
  //After all that looping and requests make a CSV file
  deferred.resolve(dataFromResponses);
  return deferred.promise();
}

// and call it with:

$.when(getDataFromWS()).done(function(dataFromResponses){
  console.log(dataFromResponses);
  // buildCSVfile(dataFromResponses);
}).fail(function(response){
  console.error(response);
});

<script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>

It works perfectly on your code snippet, but in the console.log(dataFromResponses) at the .done function it just prints like the first response id: i, if I put a console.log(dataFromResponses) before the return, it prints the whole array with the 31 responses... — Hernán Casillas, Oct 14 '20 at 16:58
thanks for your answer, so with this code I won't get the "Too many requests" error while calling the API a lot of times? Or with this code I am able to use a delay which is what makes sense to me to not get that error. — Hernán Casillas, Oct 14 '20 at 17:09
This code will handle the request in a "synchronous" way. Therefore the calls will not run simultaneously. The amount of calls one can make in a certain amount of time could still be limited by the server. Experiment a little bit with your server/backend and see if it runs into any problems. — Ashwin, Oct 14 '20 at 17:13

Dmitriy Mozgovoy · Answer 2 · 2020-10-14T19:17:46.917

I believe this is a challenge for backend- its API should support doing this task by one request. But if you're sure that you need to make many separate requests, just use concurrency limit (Live demo):

import CPromise from "c-promise2";
import $ from "jquery";

async function getDataFromWS(url, concurrency) {
  return CPromise.all(
    function* () {
      for (let i = 1; i < 32; i++) {
        const uriData = { date: "2020-08-0" + i };
        yield $.ajax({
          data: uriData,
          url,
          type: "GET",
          dataType: "json",
          async: true
        });
      }
    },
    { concurrency }
  );
}

const buildCSVfile = async (responses) => {
  console.log(`Done: `, responses.map(JSON.stringify));
};

(async () => {
  const data = await getDataFromWS(
    `https://run.mocky.io/v3/753aa609-65ae-4109-8f83-9cfe365290f0?mocky-delay=1s`,
    10 // concurrent request limit 
  );
  await buildCSVfile(data);
})();

How to handle a lof of Ajax requests on a loop and prevent "Too many requests" using JavaScript

2 Answers2