1

I am implementing NodeJS based scripts for communicating to couchbase and another service. It is a long running script and after a while I get "connect EMFILE" for the service. My code sample is given below:

function createContainer(chunkName,recordingID,chunkData)
{
  var swiftHTTPPath='http://'+swiftIPAddr+'/swift/v1/'+recordingID;
  var path = '/swift/v1/'+recordingID;
  var swiftOptions = {
    hostname : swiftIPAddr,
    port     : swiftPort,
    path     : path,
    method   : 'PUT',
  };

  http.get(swiftHTTPPath, function(res) {
    if(res.statusCode == '404')
    {
      var req = http.request(swiftOptions, function(res)
      {
        res.setEncoding('utf8');
        res.on('data', function (chunk)
        {
          console.log('Container created');
        });
        res.on('end', function() {
        });
      });
      req.on('error', function(e)
      {
        console.log(e.message);
      });
      req.end();
    }
    else
    {
      console.log('Container already exists');
    }
  }).on('error', function(e)
  {
    console.log('e.message);
  });
}

The following command shows there are more than 1024 open connections

lsof -i -n -P > mylog.log

........
node    14135 ubuntu 1020u  IPv4 5249347      0t0  TCP 10.1.1.1:53623->10.1.1.2:8091 (ESTABLISHED)
node    14135 ubuntu 1021u  IPv4 5249350      0t0  TCP 10.1.1.1:42021->10.1.1.2:11210 (ESTABLISHED)
node    14135 ubuntu 1022u  IPv4 5249351      0t0  TCP 10.1.1.1:53627->10.1.1.2:8091 (ESTABLISHED)
........

I can increase ulimit values but I would like to know am I doing the HTTP GET,POST request in the correct way. Is there a way to reuse/ terminate the connection ?

cucucool
  • 3,777
  • 8
  • 48
  • 63

1 Answers1

0

Since it looks like all (or most) of your requests go to a single host, you can set the maximum number of concurrent sockets per host to a limited number, such as 100. Do this with the agent.maxSockets option in Node's http module.

Regarding simply reusing connections, this should happen automatically. The fact that it isn't leads me to believe you are issuing many requests concurrently...so you could also solve this problem by issuing requests more gradually.

John Zwinck
  • 239,568
  • 38
  • 324
  • 436
  • Thanks for your response. Unfortunately, I cannot issue request gradually due to the nature of the system. So does that mean I am issuing 1024 simultaneous requests (I suspect that ) ? Is there something that I could have done differently in my HTTP requests ? – cucucool Sep 03 '15 at 04:05
  • My understanding was the socket will be closed once the request (GET,PUT) is complete or at least it should be re-used. I am planning to run the script 24x7 with connections to multiple systems. How do I handle this scenario ? – cucucool Sep 03 '15 at 06:47
  • @KarthikJ: Well, you could simply increase your ulimit. – John Zwinck Sep 04 '15 at 01:51