Resolving a promise when wanted

Question

I want to constantly sync data from a website, however I only have 300 calls/15 minutes. Thus I thought i could put all my sync requests (around 1000) into an array and then resolve just 300 of them every 15 minutes till the requests array is empty and then start again. However when I do the following:

  let requests = []
  params = 'invoice-items?invoice_id='
  let positions = null
  for (const invoice of invoices) {
    requests.push(new Promise(async (resolve, reject) => {
      positions = await getBillomatData(params + invoice.id, null, 0, null)
      await updateDatabase(positions, models.billomat.Position)
    }))
  }

  console.log(requests[0])
  await requests[0]
  console.log(requests[0])

As soon as I wait for the request at requests[0] it executes all of the requests and I go over the limit of calls.

[Never pass an `async function` as the executor to `new Promise`](https://stackoverflow.com/q/43036229/1048572)! — Bergi, Jan 04 '22 at 18:02
A promise executor runs immediately, regardless whether, where, and when you await the promise. Just don't construct the promises until you need them, and wait (sleep) in the loop itself instead. — Bergi, Jan 04 '22 at 18:03
As @Bergi mentions you have to construct your promises asynchronously. In other words you have establish a promise once in every 3 seconds. Then it might or might not get more convoluted depending on what you expect. Some promises may take less than 3 seconds to resolve and some may take longer. So responswise overlaps in the time scale might occur like getting an older state after a more fresh one from the server which is another question basically. — Redu, Jan 04 '22 at 18:13

Nidhin David · Answer 1 · 2022-01-04T18:22:04.263

0

The most simplest approach (not optimised though) would be

make batches of 300 calls
execute one batch and wait for all of them to be resolved before proceeding to next batch

let batch = []
// imagine urls is a array of url of 900 items
urls.map(async (url)=>{
  
  batch.push(somePromiseFuctionToDoTheApiCall(url))
  if(batch.length >= 300){
    await Promise.all(batch)
    // sleep is a promisified settimeout function, ref: https://stackoverflow.com/a/56520579/3359432
    await sleep(calculateTimeToWaitBeforeProceedingToNextBatch)
    batch = []
  }

})
// there might be some leftovers at the end of batch you should process them also

if you are ok with using libraries and stop reinventing the wheel then have a look at lodash.chunk, bluebird.map, bluebird.each, bluebird.delay

edited Jan 04 '22 at 18:22

answered Jan 04 '22 at 18:16

Nidhin David

2,426
3
31
45

The issue is, the code is running in one script of a whole website, so if I sleep till the next 15 minutes. I cant use the website during that time. – Robin Jehn Jan 04 '22 at 18:18
Please see the reference link, it is a promisified settimeout function. it is non blocking! – Nidhin David Jan 04 '22 at 18:20
@RobinJehn That's not true. You can still use the website while the request is sleeping – slebetman Jan 05 '22 at 09:34

kevintechie · Accepted Answer · 2022-01-05T09:16:59.763

All async calls are executed at once because JavaScript executes ALL code as soon as each call is processed. Await awaits the result, it does not await execution.

You'll need to use a tool such as bottleneck to rate limit your requests.

Processing 300 requests of 1000 every 15 minutes will take an hour to complete. This is a long time to keep the node job running doing nothing.

A basic a basic limiter might keep track of all the requests you want to make in an external file or database and then use a cron job to execute your JavaScript code every 15 minutes to process another 300 requests. If there are no additional requests, your app could just terminate. However, it will wake up and run every 15 minutes as long as the cron job keeps running.

Thank you very much, I tried bottleneck and it works like a charm — Robin Jehn, Jan 04 '22 at 19:45

Resolving a promise when wanted

2 Answers2