Nodejs request ETIMEDOUT when sending a lot of requests

Question

I am sending every 50ms a request (so 20requests/second) to get Body of site, but after a 20-30seconds of execution I get error ETIMEDOUT in every request. I have set process.env.UV_THREADPOOL_SIZE = 128;

I have tried setting THREADPOOL_SIZE to larger values than 128.

    for(let i in urls) {
        setTimeout(function() {
            getBody(i); //It is function with a request
        },50*i);
    }

    request({
        url:url,
        method:'GET',
        timeout:3000,
        headers: {
            'Accept': '*/*',
            'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36'
        }},
        (err,res) => {

package: request

Please help me, I want send high amount of requests in one second.

You may be overloading the server, or getting throttled for making too many requests. — Joe Clay, Mar 26 '19 at 17:48
Show `getBody`, we need to see how you're issuing the request, which module, what arguments you're passing, what's the timeout set for the request. There are multiple reasons why you may be getting that error. Also show the stack trace, because it can be a connection timeout, or a read timeout. — Marcos Casagrande, Mar 26 '19 at 17:48

Marcos Casagrande · Answer 1 · 2023-04-27T14:46:00.527

UV_THREADPOOL_SIZE is limited to 128, so you can't increase it pass that number.

Its default size is 4, but it can be changed at startup time by setting the UV_THREADPOOL_SIZE environment variable to any value (the absolute maximum is 128).

Second, there's no timeouts property in request module, the correct property name is: timeout.

If you're running multiple requests in parallel to the same server, setting it to 3000 may be too low, specially if the server can't handle that many requests.

That may be one of the reasons you're getting so many timeouts error. So first of all increase that number to see if you still get timeouts.

Second, there are two different type of timeouts:

There are two main types of timeouts: connection timeouts and read timeouts. A connect timeout occurs if the timeout is hit while your client is attempting to establish a connection to a remote machine (corresponding to the connect() call on the socket). A read timeout occurs any time the server is too slow to send back a part of the response.

You can check if the timeout was a connection timeout doing the following:

if(err.connect === true) // connection timeout.

Also, if you're hitting the same domain, to increase the speed & reduce the number of timeouts, you can resolve the IP address, and hit the server directly using the IP.

UPDATE: see https://stackoverflow.com/a/73581100/1119863

I didn't test the script, but it's to show you how you would do it, a minor fix may be needed

const dns = require('dns');
const { promisify } = require('util');
const URL = require('url');
const dnsResolve = promisify(dns.resolve);


const ipCache = {};

async getIP(host) {
    
    // Use LRU cache with expiration...

    if(ipCache[host]) // If it's set, it's either the resolved IP or dnsResolve Promise
        return ipCache[host];

    ipCache[host] = dnsResolve(host);

    const [ip] = await ipCache[host]; // Wait until the promise is resolved

    ipCache[host] = ip;

    return ip;
}

async getBody(url) {
    
    const { host } = new URL(url);


    const ip = await getIP(host);

    // Build the URL using the IP address instead of the domain
    request( /* ... */)
}

You should also throttle the requests, because you may be flooding your own network. If you try this on a VPS, you're probably going to get less timeouts than running the script on your local machine.

For example on AWS there are some network optimized instances that are better suited for making a lot of requests, using one of those instances will allow you to make more successful requests than on your local machine.

So either it's a limitation of your own network, or a limitation of server where you're issuing the requests to, in both cases, you will need to throttle the requests to avoid the error.

No, I just changed library to axios and this problem doesn't occur in this library. — ksew, Mar 29 '19 at 20:47

Nodejs request ETIMEDOUT when sending a lot of requests

1 Answers1