Many resquest on GetStaticProps freezes my application build and return an server error

Question

I'm trying to list a bunch of products and I wanted to request data on node and build the page in a static way, so The homepage would be faster.

The problem is that when I make over 80 request on GetStaticProps.

The following code with 80 items, does work

const urlList = [];

for (let i = 1; i <= 80; i++) {
  const url = `myApiUrl`;
  urlList.push(url);
}

const promises = urlList.map(url => axios.get(url));
const responses = await Promise.all(promises);

return responses;

The following code with 880 items, does not work (Note that is does work outside of GetStaticProps))

const urlList = [];

for (let i = 1; i <= 880; i++) {
  const url = `myApiUrl`;
  urlList.push(url);
}

const promises = urlList.map(url => axios.get(url));
const responses = await Promise.all(promises);

return responses;

erro on console:

Uncaught     at TLSWrap.onStreamRead (internal/stream_base_commons.js:209:20)

webpage error:

Server Error
Error

This error happened while generating the page. Any console logs will be displayed in the terminal window.

TLSWrap.onStreamRead
internal/stream_base_commons.js (209:20)

Is there a way to handle large requests amount like that? I'm new to hhtp requests, is there a way for me to optimize that?

Rather than doing `Promise.all` on all of the requests, you could either await one after the other, or batch the requests (into groups of maybe 10 or 20) and run those batches into `Promise.all`, then continue with the next batch, and so on. — Chris Gilardi, Jun 21 '21 at 18:17
Sadly, trying to request small batches didn't really solve the crash. Await one by one does work — Sku, Jun 21 '21 at 22:44
Do you _absolutely_ have to make that many requests? Couldn't the API provide an endpoint to return X amount of products on a single request instead? — juliomalves, Jun 23 '21 at 16:40
I have no control over the api and there is no endpoint that returns a list with everything that I need, I would have to create my own api to do that — Sku, Jun 24 '21 at 02:15
Could you show how exactly you're trying to batch this, are you absolutely sure those requests are not still executed in parallel? You can easily check this by reducing the size of 'chunk' to 1; it should behave exactly the same as in waterfall case. — raina77ow, Jun 29 '21 at 10:44
Also, @juliomalves is right: there's no way the system that attempts to do 800 API calls to render a single page scales well. Perhaps you need to store at least some responses in cache? — raina77ow, Jun 29 '21 at 10:45

score 3 · Accepted Answer · answered Jun 30 '21 at 08:26

There are limits to how many connections you can create to fetch content. What you're seeing is that a method like Promise.all() isn't "smart" enough to avoid running into such limits.

Basically, when you call Promise.all() you tell the computer "do all these things simultaneously, the order does not matter, and give me all the output when done. And by the way, if a single of those operations fail stop everything and throw away all other results". It's very useful in many contexts, but perhaps not when trying to fetch over 800 things from the net..

So yes, unless you can tweak the requirements like number of allowed simultaneous connections or memory the script gets to use, you'll likely have to do this in batches. Perhaps one Promise.all() for slices of 100 jobs at a time, then next slice. You could look at using the async library and the mapLimit method or roll your own way to slice the list of jobs into batches.

I'm having a lot of trouble to make the batches slices to work with getStaticProps, but I think your answer is in the right path. — Sku, Jul 04 '21 at 08:02

score 1 · Answer 2 · answered Jun 26 '21 at 18:31

1

this could be a problem based on the node version its using

but for await could also be an option for you...

answered Jun 26 '21 at 18:31

Noriller

342
1
4
12

score 1 · Answer 3 · answered Jun 29 '21 at 10:41

1

You can leverage axios.all instead of Promise.all.

const urlList = [];

for (let i = 1; i <= 80; i++) {
  const url = `myApiUrl`;
  urlList.push(url);
}

const promises = urlList.map(url => axios.get(url));
const responses = await axios.all(promises);

return responses;

https://codesandbox.io/s/multiple-requests-axios-forked-nx1z9?file=/src/index.js

answered Jun 29 '21 at 10:41

tperamaki

1,028
1
6
12

1

this doesn't really make any difference in my code right now – Sku Jul 04 '21 at 08:00
Ok, it seems to work fine on the codesandbox, so the issue must be something else – tperamaki Jul 04 '21 at 09:47

score 1 · Answer 4 · answered Jul 02 '21 at 09:41

As for step, for debugging purposes I would use Promise.allSettled instead of Promise.all. This should help you to understand what is the error returned by the HTTP socket. If you don't control the external API, it is likely that a firewall is blocking you from this "DDOS" attack.

As you said, batching the call doesn't solve the issue (if you queue 80 requests followed by 80 etc, you may encounter the rate limit in any case)

You should check for throttling issues, and use a module to speed limit your HTTP call like throttle-debounce

Many resquest on GetStaticProps freezes my application build and return an server error

4 Answers4