Parallel Fetch With Limit

Question

Currently using

async function getAllUrls(urls) {
try {
    var data = await Promise.all(
        urls.map(
            url =>
                fetch(url).then(
                    (response)
                )));

    return (data)

} catch (error) {
    console.log(error)

    throw (error)
}}

But urls is often array of 20+ links which doesnt make the api rate limiter very happy, what am looking for is way to limit how many requests it can send at once, for example limit it to 5 requests at once, after its done go for the other 5

https://stackoverflow.com/questions/67434001/javascript-split-requests-into-chunks-with-a-promise-all — Andreas, Aug 26 '22 at 11:46
But if a rate limiter is the problem, how long should you wait after sending a grap of 5? Also when error gets thrown, is it ever a message indicating that you're being rate limited? I ask that second question because if that is the case, you can set behaviour on it — The Bomb Squad, Aug 26 '22 at 12:27
like 200ms and if we hit the rate limit its already late, need to prevent hitting it — cidodo7472, Aug 26 '22 at 12:38

score 0 · Answer 1 · answered Aug 26 '22 at 11:46

0

Hey there you can use slice() function of js.. For example-

urls.slice(0,5).map(.......

in the slice() function, 0 is the starting point of your data array while 5 is the ending. You can set 5 to anything that you like. If you got any issue just lemme know. I will try to help you..

answered Aug 26 '22 at 11:46

Varun Kaklia

366
4
9

How to get the next chunk? How to make that "rate limiter" happy/-ier? This is just a comment... – Andreas Aug 26 '22 at 11:47

score 0 · Accepted Answer · answered Aug 26 '22 at 12:16

A naive approach would be to fetch them in batches, waiting the rate limit period before requesting the next batch:

const sleep = (ms) =>
  new Promise((resolve) => {
    setTimeout(resolve, ms);
  });

/**
 * @param {string[]} urls
 * @param {number} delayInterval
 * @param {number} batchSize
 */
const fetchInBatches = async (urls, delayInterval, batchSize) => {
  const remaining = [...urls];

  const responses = [];

  while (remaining.length !== 0) {
    const batch = remaining.splice(0, batchSize);

    const [batchResponses] = await Promise.all([
      Promise.all(batch.map((url) => fetchUrl(url))),
      sleep(delayInterval),
    ]);

    responses.push(...batchResponses);
  }

  return responses;
};

A better and technically more challenging approach is to write a rate-limited request function that can be used to make any arbitrary request to the rate-limited API.

Parallel Fetch With Limit

2 Answers2