Javascript | Split requests into chunks with a promise.all

Question

We are trying to split a request up into chunks as an external API has a limit on the number of products we can display per page.

Say we have a total of 113 products, but only 5 are displayed per page, which are fetched via passing in the product id as a parameter (productIds[]=x&productIds[]=y). We know there's a total of 113 and a limit of 5, however, we don't want to slow this down by waiting for the previous request to finish, so we would like to chunk this using a Promise.all().

I know I can use slice for this, however, I was hoping that I would just be able to map it into an array.

So we have a start array which is like

[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, ...]

These are all of the product ids, we then need to send a request per 5 products referencing the ids.

await axios.get('http://external.api/product', { params: { productIds: productIds.slice(0, 5) } }

However, I would like to do something like the following:

Promise.all(
  productIds.map(
    product => axios.get('...', { params: {productIds: (subset of 5 products in chunks )}
  )
)

Use one of the answers [here](https://stackoverflow.com/questions/8495687/split-array-into-chunks) and write `Promise.all(chunks(productIds).map(chunk => ...` — georg, May 07 '21 at 11:25

score 0 · Answer 1 · answered May 07 '21 at 11:31

I would use two functions to achieve this. chunk with promiseAll or delay and chunk.

Here is the chunk function I use to group list by count.

const chunk = <T>(arr: T[], size: number): T[][] => [
  ...Array(Math.ceil(arr.length / size)),
].map((_, i) => arr.slice(size * i, size + size * i));

And delay function with Promise.

const delay = (ms: number) => {
  return new Promise<void>((resolve) => setTimeout(resolve, ms));
}

If the API doesn't have any limitations time-wise you can use this:

const groupedIdList = chunk(idList, PAGESIZE);
const result = await Promise.all(groupedIdList.map(idList) => myFetchFunctionPromise(idList));
const data = result.flat();

If the API has some limits you need to delay your requests.

  const data = await groupedIdList.reduce(async (prev, subIdList) => {
    const result = await prev;
    const newDataList = await myFetchFunctionPromise(subIdList);
    // wait some time ms
    await delay(500);
    return [...result, ...newDataList];
  }, Promise.resolve([]));

Javascript | Split requests into chunks with a promise.all

1 Answers1

Linked