33

I'm writing a simple api endpoint to determine if my server is able to reach the internet. It works great, but after 5 requests (exactly 5, every time) the request hangs. Same thing happens when I switch Google to Hotmail.com, which makes me think that this is something on my end. Do I need to close the http.get requests? I was under the impression that this function closes the requests automatically.

// probably a poor assumption, but if Google is unreachable its generally safe to say     that the server can't access the internet
// using this client side in the dashboard to enable/disable internet resources

app.get('/api/internetcheck', function(req, res) {
console.log("trying google...");
    http.get("http://www.google.com", function(r){
        console.log("Got status code!: " +r.statusCode.toString());
        res.send(r.statusCode.toString());
        res.end();
        console.log("ended!"); 
    }).on('error', function(e) {
        console.log("Got error: " + e.message);
    });
});
scald
  • 729
  • 1
  • 7
  • 10
  • Having the "res.end()" line appears to have the browser "hang" (i.e. Chrome would be stuck loading the page). Deleting would result in the status code being rendered successfully (no matter how many times or how quickly those requests were made - I tried 100 times in a for loop and all went well). – funseiki Jun 06 '13 at 16:06

2 Answers2

54

Here's the reason of the "exactly 5": https://nodejs.org/docs/v0.10.36/api/http.html#http_agent_maxsockets

Internally, the http module uses an agent class to manage HTTP requests. That agent will, by default, allow for a maximum of 5 open connections to the same HTTP server.

In your code, you don't consume the actual response sent by Google. So the agent assumes that you're not done with the request, and will keep the connection open. And so after 5 requests, the agent won't allow you to create a new connection anymore and will start waiting for any of the existing connections to complete.

The obvious solution would be to just consume the data:

http.get("http://www.google.com", function(r){
  r.on('data', function() { /* do nothing */ });
  ...
});

If you run into the problem that your /api/internetcheck route is called a lot, so you need to allow for more than 5 concurrent connections, you can either up the connection pool size, or just disable the agent completely (although you would still need to consume the data in both cases);

// increase pool size
http.globalAgent.maxSockets = 100;

// disable agent
http.get({ hostname : 'www.google.com', path : '/', agent : false }, ...)

Or perhaps use a HEAD request instead of GET.

(PS: in case http.get generates an error, you should still end the HTTP response by using res.end() or something like that).

NOTE: in Node.js versions >= 0.11, maxSockets is set to Infinity.

robertklep
  • 198,204
  • 35
  • 394
  • 381
  • 2
    Just to add to this excellent answer - my issue was when I loaded PNG blobs from a remote server, in the instance there wasn't a matching blog I defaulted to a default binary blob instead of returning a 404 error. I got this issue because I wasn't consuming the loaded data... very clever! So as above - include r.on('data', function() { /* do nothing */ }); and it tricks it into thinking you do and closes the connection! – sidonaldson Oct 22 '13 at 16:04
  • Reviving a fairly old post, but, does the consumption requirement also apply to non-200 responses? If you get something like a 4xx or 5xx and don't consume using r.on('data'), is that connection not closed? – Johnny Dec 16 '14 at 19:19
  • 1
    @Johnny I guess it would depend (somewhat) on the remote server, but even for non-200 responses I think you should assume that you need to read the entire response before the `http` module will return the connection to the pool. – robertklep Dec 17 '14 at 08:05
  • @robertklep yup, you were right, I added something to consume on bad responses too and the hangups stopped! – Johnny Dec 18 '14 at 09:23
  • @matt the light was seen and recent Node versions set a limit of `Infinity`. – robertklep Jun 09 '15 at 14:50
  • Good to know! didn't realize I haven't upgraded in a while! nonetheless it's still easy to miss on consuming the data and leak connections, unless the api changed that too.... – matanster Jun 09 '15 at 15:22
3

If you wait long enough, the 5 requests will time-out and the next 5 will process so the application isn't really hanging, because it will eventually process through all of the requests.

To speed up the process you need to do something with the response data such as r.on('data', function() {});

Bryan Focht
  • 1,004
  • 1
  • 11
  • 10