Do too much requests cause errors in nodejs?

Question

var urlArr = {
    //ex url_for_site0 = 'https://www.google.com'
    url_for_site0,
    url_for_site1,
    url_for_site2,
    url_for_site3,
    ...
    url_for_site50
};

urlArr.forEach(function(url, index) {
    request(url, function(err, res, body) {
        if(err) console.log(index+err);
        else console.log(index+" success");
    });
});

I got different unordered results and errors everytime I execute my app.

Example:

1 error : socket hang out
21 error : socket hang out
17 error : socket hang out
1 error : socket hang out
19 error : socket hang out
...(omission)
5 success
15 success
45 success
50 success
11 success
37 success

Everytime I get the results, they are in a different order. Is this becasue I called too much request simultaneously? When I request one by one, there's no error.

Example:

request(url_for_site0)
and restart program
request(url_for_site1)
and restart program
request(url_for_site2)
...

Possible duplicate of [Why is node.js only processing six requests at a time?](https://stackoverflow.com/questions/12060869/why-is-node-js-only-processing-six-requests-at-a-time) — Gabriel Bleu, Apr 18 '18 at 11:35
I think it's a question for an old version. Another question&answer says I can call thounsands of request — Jinsu, Apr 18 '18 at 15:18
There is both historical and recent informations, for example you could try with `agent:false` to bypass the pool. — Gabriel Bleu, Apr 18 '18 at 15:32
I knew the reason why errors threw. Everyone's opinion was right. The error was because of simultaneous request calls. and If you don't want to use callback structure, I recommend you to use request-promise!! — Jinsu, Apr 24 '18 at 02:38

score 1 · Answer 1 · edited Apr 18 '18 at 20:29

NodeJS events are all handled in a single pool and has a non-blocking nature. You can refer to the illustration below.

It happened to me once when I try to call multiple SQL queries. When I did it using C#, there is no problem at all. However, NodeJS gave me a similar behaviour to yours.

nodejs

I am not sure if this is the best solution for the problem. However, here is how I fixed my problem with my SQL calls. I used the async waterfall function so that the whole process becomes synchronous. Each function will be run one by one with its return value piped to the next function. So, you can even do more stuffs. The usage of this library is not very staightforward, you can refer to this link to better help you understand how async waterfall works, then suit it to fit your solution.

https://gist.github.com/dineshsprabu/e6c1cf8f2ca100a8f5ae

Here is how I visualize your solution will roughly looks like:

var async = require('async');

async.waterfall(
[
    function(callback) {
        function_urlArr(url, index, function (returnVal) {
            //Do something with the returnVal
            callback(null, returnVal);
        });

    },
    function(returnVal, callback) {
        //the returnVal from first function gets passed here synchronously
        function_urlArr(url2, index2, function (returnVal) {
            //Do something with the returnVal
            callback(null, returnVal);
        });
    },
    function(returnVal, callback) {
        //and so on ...
    }
],
function (err) {
    //console.log(err);
});

//define your function and enable callback
//you will need to include an extra third argument to receive the callback
function urlArr(url, index, callback) {
    //your code
    return callback(returnValue)
}

score 0 · Answer 2 · answered Apr 18 '18 at 11:36

0

This is happening because of non-blocking nature of javascript.

If you want to make it happen one by one in order, you can use Async functions.

Socket hung up error may be because of the url you hit didn't respond anything after accepting the request.

answered Apr 18 '18 at 11:36

krishnashu

42
5

Everytime I run the app, the urls which cause socket hang up errors changes. Example) urlArr[1] caused a socket hang up error but when I restart the app, urlArr[1] don't cause a socket hang up error.. Is this because the server didn't repond to me?? – Jinsu Apr 18 '18 at 12:03

Sumit Jha · Answer 3 · 2018-04-18T21:49:56.200

You might have issue with non-blocking nature of loop forEach. You can combine Promise and aysnc/await to make it blocking. Here is one way of handling it.

const request = require('request');

let urlArr = [
    'https://localhost:9090',
    'https://www.google.com',
    'https://www.ebay.com',
    'https://www.amazon.com',
];

//Creating promise for the requests.
let fetchPromise = function(url) {
    return new Promise((resolve, reject) => {
            request(url, (err, res, body) => {
                if (err)
                    reject(Error(url + ' cannot be fetched'));
                else
                    resolve(body);
            });
        }
    );
};

//creating a blocking function
let fetchAllData = async function(urls) {
    for (url of urls) { //using modern for loop instead for forEach
        try {
            data = await fetchPromise(url); // waiting until promise is resolved.
            console.log('Recieved :' + data.length + 'bytes from ' + url);
        } catch(e) {
            console.log('Error :' + e); // catching error in case promise is rejected
        }              
    }
};

//calling the function
fetchAllData(urlArr);

/*
// In case you want to wait until all promises are resolved.
// Then use Promise.all, however it will fail if any of the promise is rejected.
// One way to handle it would be to modify function fetchPromise such that it 
// always resolves.

Promise
.all(urlArr.map(url => fetchPromise(url)))
.then(data => console.log(data))
.catch(err => console.log(err));
*/

I hope it helps.

Do too much requests cause errors in nodejs?

3 Answers3