Sleep / delay inside promise.all

Question

I am building a backend to handle pulling data from a third party API. There are three large steps to this, which are:

1. Delete the existing db data (before any new data is inserted)
2. Get a new dataset from the API
3. Insert that data.

Each of these three steps must happen for a variety of datasets - i.e. clients, appointments, products etc.

To handle this, I have three Promise.all functions, and each of these are being passed individual async functions for handling the deleting, getting, and finally inserting of the data. I have this code working just for clients so far.

What I'm now trying to do is limit the API calls, as the API I am pulling data from can only accept up to 200 calls per minute. To quickly test the rate limiting functionality in code I have set it to a max of 5 api calls per 10 seconds, so I can see if it's working properly.

This is the code I have so far - note I have replaced the name of the system in the code with 'System'. I have not included all code as there's a lot of data that is being iterated through further down.

let patientsCombinedData = [];
      let overallAPICallCount = 0;
      let maxAPICallsPerMinute = 5;
      let startTime, endTime, timeDiff, secondsElapsed;

      const queryString = `UPDATE System SET ${migration_status_column} = 'In Progress' WHERE uid = '${uid}'`;

      migrationDB.query(queryString, (err, res) => {
        async function deleteSystemData() {
          async function deleteSystemPatients() {
            return (result = await migrationDB.query("DELETE FROM System_patients WHERE id_System_account = ($1) AND migration_type = ($2)", [
              System_account_id,
              migrationType,
            ]));
          }

          await Promise.all([deleteSystemPatients()]).then(() => {
            startTime = new Date(); // Initialise timer before kicking off API calls

            async function sleep(ms) {
              return new Promise((resolve) => setTimeout(resolve, ms));
            }

            async function getSystemAPIData() {
              async function getSystemPatients() {
                endTime = new Date();
                timeDiff = endTime - startTime;
                timeDiff /= 1000;
                secondsElapsed = Math.round(timeDiff);

                if (secondsElapsed < 10) {
                  if (overallAPICallCount > maxAPICallsPerMinute) {

                    // Here I want to sleep for one second, then check again as the timer may have passed 10 seconds
                    getSystemPatients();
 
                  } else {
                    // Proceed with calls
                    dataInstance = await axios.get(`${patientsPage}`, {
                      headers: {
                        Authorization: completeBase64String,
                        Accept: "application/json",
                        "User-Agent": "TEST_API (email@email.com)",
                      },
                    });

                    dataInstance.data.patients.forEach((data) => {
                      patientsCombinedData.push(data);
                    });
                    overallAPICallCount++;
                    console.log(`Count is: ${overallAPICallCount}. Seconds are: ${secondsElapsed}. URL is: ${dataInstance.data.links.self}`);

                    if (dataInstance.data.links.next) {
                      patientsPage = dataInstance.data.links.next;
                      await getSystemPatients();
                    } else {
                      console.log("Finished Getting Clients.");
                      return;
                    }
                  }
                } else {
                  console.log(`Timer reset! Now proceed with API calls`);
                  startTime = new Date();
                  overallAPICallCount = 0;
                  getSystemPatients();
                }
              }

              await Promise.all([getSystemPatients()]).then((response) => {
                async function insertSystemData() {
                  async function insertClinkoPatients() {
                    const SystemPatients = patientsCombinedData;

Just under where it says ' if (secondsElapsed < 10) ' is where I want to check the code every second to see if the timer has passed 10 seconds, in which case the timer and the count will be reset, so I can then start counting again over the next 10 seconds. Currently the recursive function is running so often that an error displayed related to the call stack. I have tried to add a variety of async timer functions here but every time the function is returned it causes the parent promise to finish executing.

Hope that makes sense

Answer 1

I ended up using the Bottleneck library, which made it very easy to implement rate limiting.

const Bottleneck = require("bottleneck/es5");

      const limiter = new Bottleneck({
        minTime: 350
      });

await limiter.schedule(() => getSystemPatients());