Node Server crashes after few hours

Question

I am using Nodemon with Forever Module on Ubuntu Server.

I use this command to start my Node Server:

forever start -c nodemon app.js  --exitcrash

It works fine for few hours (approx 48 Hours), but after that my Server stops working with these errors:

Error: getaddrinfo EMFILE   
TypeError: Cannot call method 'indexOf' of undefined
Error: Handshake inactivity timeout

These errors are caused due to Exceeding Limit of Open Files/Sockets.

Now my question is:

Can I use -m (Which sets to unlimited in my Operating System):

max memnory size   (kbytes, -m) unlimited

Should I use the above command with -m? Are there any drawbacks?

Or is there any other efficient solution to fix Server crashing?

check those answers http://stackoverflow.com/questions/410616/increasing-the-maximum-number-of-tcp-ip-connections-in-linux http://serverfault.com/questions/48717/practical-maximum-open-file-descriptors-ulimit-n-for-a-high-volume-system — Edgar Zakaryan, Apr 09 '15 at 12:22
@AleksandrM i have large number of users in my site i have checked that node server closes the sockets itself . socket.on('disconnect', function () { var socketIndex = connectionsArray.indexOf( socket ); console.log('socket = ' + socketIndex + ' disconnected'); if (socketIndex >= 0) { connectionsArray.splice( socketIndex, 1 ); } console.log(socketIndex + 'Number of connections:' + connectionsArray.length); }); console log shows number of currently used open sockets . — Irtiza shahid, Apr 09 '15 at 12:29
@EdgarZakaryan is there any drawback if i use # ulimit -n 99999 command to increase the number of files limit on my server ? — Irtiza shahid, Apr 09 '15 at 12:34
@EdgarZakaryan what will happen if i use command forever start -c nodemon app.js --exitcrash with -m instead of -C ? can i use this ? is there any drawback ? because max memnory size is already sets to unlimited in my server — Irtiza shahid, Apr 09 '15 at 12:40

score 2 · Answer 1 · edited May 23 '17 at 11:44

2

If you have large number of users most probably you are hitting systems maximum number of requests queued to listen socket. If you are sure your server can handle the load you can increase from default 128 to something 1024.

And yes, increase the ulimit, so system can handle more load, but don't set to unlimited, just check what is enough to handle current load.

Also go through this Increasing the maximum number of tcp/ip connections in linux will get some helpful info too

edited May 23 '17 at 11:44

Community

1
1

answered Apr 09 '15 at 12:49

Edgar Zakaryan

576
5
11

my current open files limit is 2048 open files (-n) 2048 – Irtiza shahid Apr 09 '15 at 13:10
@Irtizashahid you can double it – Edgar Zakaryan Apr 09 '15 at 13:16
i have increase the value of (-n) open values . i have to wait for min two days to test the server .you have any idea what will happen if i change -c to -m on my linux command ? – Irtiza shahid Apr 10 '15 at 05:19
@Irtizashahid you want to pass -m to forever? -m MAX Only run the specified script MAX times – Edgar Zakaryan Apr 10 '15 at 05:24
still same issue :( node server stop working .I have increased the value of open files (-n). – Irtiza shahid Apr 11 '15 at 16:00
@Irtizashahid if it's taking over 2 days for your server to crush, there is possibility you have some leaks, maybe you are not closing opened connections – Edgar Zakaryan Apr 11 '15 at 18:08
Dear i have implemented 'disconnect' event on my node server in console it displays only open sockets you can see my disconnect code ....... socket.on('disconnect', function () { var socketIndex = connectionsArray.indexOf( socket ); console.log('socket = ' + socketIndex + ' disconnected'); if (socketIndex >= 0) { connectionsArray.splice( socketIndex, 1 ); } console.log(socketIndex + 'Number of connections:' + connectionsArray.length); }); – Irtiza shahid Apr 11 '15 at 19:09
@Irtizashahid can you check if socket heartbeat is enabled, it should be enabled so if there is no activity in the connection, it gets closed automatically – Edgar Zakaryan Apr 12 '15 at 12:18

score 0 · Answer 2 · answered Apr 15 '15 at 21:14

This is probably not the ideal answer but using forever-service with nodemon will ensure your server restarts after it crashes.

This is the command that worked for me. I'm including it because getting forever-service and nodemon to play well can be tricky.

It does the following: everytime a json or raml file in the applications dist/assets folder is modified, wait 10 seconds and then restart the node app (server.js script):

$ forever-service install raml --script server.js -f " -c nodemon" -o " --delay 10 --watch dist/assets -e json,raml --exitcrash" -e "PATH=/usr/local/bin:$PATH"

Then I can run:

$ service raml start|stop|restart|status

I can also have the service start on server reboot with the chkconfig utility:

$ chkconfig --add raml
$ chkconfig raml on

The issue on my server is limit exceeded if number of sockets increases on my server .can it be resolved if i use service raml or chkconfig ? i have no idea about rmal and chkconfig service — Irtiza shahid, Apr 16 '15 at 07:49
I'm sorry for the confusion. This is just a method of how you can restart the server automatically when it crashes. It does not address your original error. The rest of the information is included to show you how to create the service since it can be difficult. — Julia Anne Jacobs, Apr 17 '15 at 12:04

Node Server crashes after few hours

2 Answers2

Linked