Dealing with rate limiting APIs in NodeJS

Question

So I need a way to run a function x amount of times in a given second, then wait until the next second to run the next set. I'm using the Yelp Fusion API to call a 'search' https://api.yelp.com/v3/search and then running a details query on each of the search results: https://api.yelp.com/v3/businesses/business_id.

So the first search query returns an array of businesses like this:

const businesses = response.body.businesses
for (business of businesses) {
    fetchPlaceDetails(business.id)
}

Theoretically I can run fetchPlaceDetails() in an async manner, but going through 50+ results takes too long. Instead, I'd like to run the query maybe 5 times, wait a second (to get past the rate limit), run the next 5, wait a second, run the next 5, etc.

Not sure how to implement this but I figure it's got to be a pretty standard issue to have when using APIs with rate limits. Also, I'm not sure but nowhere in the documentation do I see an actual call/second limit specified, but I'd assume it's maybe 10 requests/second or so?

What do you mean "*I can run `fetchPlaceDetails()` in an async manner*"? Surely it already is asynchronous, no? Does it return a promise? — Bergi, Jul 09 '22 at 17:19
Have you searched for rate limiting amywhere? There's hundreds of examples on stackoverflow alone — Jaromanda X, Jul 09 '22 at 17:20
@Bergi I mean I can slap an async in front of it and run them one at a time with no rate limiting issues but it takes a while when you're iterating through 50+ options and waiting for each previous request to finish before starting the next one — nickcoding2, Jul 09 '22 at 17:24
@JaromandaX I have, yeah. Just not entirely sure how it works in this simple case. — nickcoding2, Jul 09 '22 at 17:25
@nickcoding2 You mean slapping `await` in front of it? So that they execute sequentially? That might actually still run into rate limiting, if each request takes less than 200ms you'd have more than 5 per second. — Bergi, Jul 09 '22 at 17:27
Array.splice and Promise.all could help. Along with a delay. What is the rate limit — Jaromanda X, Jul 09 '22 at 17:27
@Bergi Yes, I meant await my bad :) Well I didn't run into any issues with it, not sure how long each request was taking but all results returned fine. But yeah it's definitely not a reliable way of getting past the rate limiting. One of the problems is I don't even know what the rate is because Yelp doesn't document it... — nickcoding2, Jul 09 '22 at 17:28
See `rateLimitMap()` [Method for batch processing max requests per second](https://stackoverflow.com/questions/36730745/choose-proper-async-method-for-batch-processing-for-max-requests-sec/36736593#36736593) for something that sounds like it would do what you want. — jfriend00, Jul 09 '22 at 17:44

score 4 · Accepted Answer · answered Jul 09 '22 at 17:31

I'd like to run the query maybe 5 times, wait a second (to get past the rate limit), run the next 5, wait a second, run the next 5, etc.

Then just do that:

const {businesses} = response.body;
for (let i=0; i<businesses.length; i+=5) {
    await Promise.all([
        ...businesses.slice(i, i+5).map(({id}) => fetchPlaceDetails(id)),
        new Promise(resolve => { setTimeout(resolve, 1000); })
    ]);
}

This will also ensure that you wait more than one second between the batches if the 5 requests take longer than that.

score 0 · Answer 2 · answered Jul 09 '22 at 18:23

Here is a "generic" method that checks if storage array contains any data, if it does, it processes nn number of items from the array then waits ii number of seconds and repeats the process.

const queue = (() =>
{
  let itemsPerStep = 5, //number of items process per step
      intervalSpeed = 5000, //how long to wait between steps in miliseconds
      stopWhenDone = true, //when queue list depleted stop the loop
      prevDate = 0,
      timer;

  const list = []; //queue list

  const commands = {
    add: item => list.unshift(item),
    get list() { return list; },
    clear: () => list.length = 0,
    speed: (val = intervalSpeed) => intervalSpeed = val,
    itemsPerStep: (val = itemNumberPerStep) => itemNumberPerStep = val,
    stop: () => (timer && console.log("stopped"), timer = clearInterval(timer)),
    start: (t = true) => (stopWhenDone = t, commands.stop(), timer = setInterval(loop), loop())
  };

  const loop = () =>
  {
    const now = new Date();
    if (now - prevDate < intervalSpeed)
      return;

    if (!list.length)
    {
      if (stopWhenDone)
        commands.stop();
      return;
    }

    prevDate = now;

    let res = []; //example result storage

    for(let i = 0; i < itemsPerStep && list.length; i++)
    {
      const value = list.pop(); //get item from the queue list

      res.push(value); //process value
    }
    
    console.log("processed ids: " + res);
  } //loop()

  return commands;
})();


//generate ids
for(let i = 100; i < 230; i += 11)
  queue.add(i);

queue.start();

Dealing with rate limiting APIs in NodeJS

2 Answers2

Linked