Auto restart or Proper Handling socket.io server on exception in linux

Question

I'ved got my socket.io server running in the background. However, there are several situation where it died. What can I do to make my socket.io server auto restart whenever it dies or terminated.

This is how I started my node server

> node server &

however the server thrown an exception and died.

events.js:141
      throw er; // Unhandled 'error' event
      ^

Error: Redis connection to pub-redis-us-east1-garantiabata.com:12087 failed - connect ETIMEDOUT 54.179.28.165:12087
    at RedisClient.on_error (/home/org-chatserver/node_modules/redis/index.js:185:24)
    at Socket.<anonymous> (/home/org-chatserver/node_modules/redis/index.js:95:14)
    at emitOne (events.js:77:13)
    at Socket.emit (events.js:169:7)
    at emitErrorNT (net.js:1253:8)
    at doNTCallback2 (node.js:450:9)
    at process._tickCallback (node.js:364:17)
^C

This is my environment

Linux version 2.6.32-042stab090.5 (root@kbuild-rh6-x64) (gcc version 4.4.6 20120305 (Red Hat 4.4.6-4) (GCC) ) #1 SMP Sat Jun 21 00:15:09 MSK 2014

and I believe my team is using this:

https://github.com/jbavari/ionic-socket.io-redis-chat

basically im forking out this https://github.com/jbavari/ionic-socket.io-redis-chat/blob/master/server.js. node -v says its v5.0.0 — Axil, Mar 26 '16 at 15:59
i tried to do this https://gist.github.com/survivaltravel/f29c38e055d0785c56ff based on https://github.com/NodeRedis/node_redis#usage, i dont know i ran it the whole day today still it died, not sure when. have to find a way to save the logging instead of in console — Axil, Mar 26 '16 at 16:02
You can use forever : http://stackoverflow.com/questions/16369018/forever-nodemon-running-together/20306929 and did you check redis connection : Error: Redis connection to pub-redis-us-east1-garantiabata.com:12087 failed - connect ETIMEDOUT 54.179.28.165:12087 — Thanh Nguyen Van, Mar 28 '16 at 06:48

Tan Hong Tat · Answer 1 · 2016-03-30T02:04:23.400

You can use Supervisor - http://supervisord.org

Supervisor is a client/server system that allows its users to monitor and control a number of processes on UNIX-like operating systems.

After installation, create a supervisor config file for your app.

For example, under /etc/supervisor/conf.d/myserver.conf -

[program:my-server]
    command=node /path/to/server.js
    numprocs=1
    autostart=true  
    autorestart=true
    startretries=99999
    environment=NODE_ENV=production  
    stderr_logfile=/var/log/my-server.err.log  
    stdout_logfile=/var/log/my-server.log  
    user=myuser

Edit the supervisor main config file - /etc/supervisor/supervisord.conf

[unix_http_server]
file=/var/run/supervisor.sock    ; (the path to the socket file)
chmod=0700                       ; socket file mode (default 0700)

[supervisord]
logfile=/var/log/supervisor/supervisord.log ; (main log file;default    ] $CWD/supervisord.log)
pidfile=/var/run/supervisord.pid            ; (supervisord pidfile;default supervisord.pid)
childlogdir=/var/log/supervisor             ; ('AUTO' child log dir, default $TEMP)

; the below section must remain in the config file for RPC
; (supervisorctl/web interface) to work, additional interfaces may be
; added by defining them in separate rpcinterface: sections
 [rpcinterface:supervisor]
supervisor.rpcinterface_factory =  supervisor.rpcinterface:make_main_rpcinterface

[supervisorctl]
serverurl=unix:///var/run//supervisor.sock ; use a unix:// URL  for a unix socket

; The [include] section can just contain the "files" setting.  This
; setting can list multiple files (separated by whitespace or
; newlines).  It can also contain wildcards.  The filenames are
; interpreted as relative to this file.  Included files *cannot*
; include files themselves.

[include]
files = /etc/supervisor/conf.d/*.conf

Usage: /etc/init.d/supervisord {start|stop|restart|force-reload|status|force-stop}

For more information/documentation - please read http://supervisord.org/

Im not sure how to create the "supervisor config file". i get this when trying to do echo_supervisord_conf https://gist.github.com/anonymous/eb524d1931dd621f051b. i found this sample, can i use this ? https://gist.github.com/didip/802561. also how do i start supervisord ? — Axil, Mar 29 '16 at 14:02
Updated with more examples - please also read the documentation on - http://supervisord.org/ — Tan Hong Tat, Mar 30 '16 at 02:05

score 0 · Accepted Answer · answered Mar 29 '16 at 14:31

0

You can use pm2 to monitor logs, auto restart... etc. For example:

sudo npm install -g pm2
pm2 start app.js --watch

More about watch & restart.

answered Mar 29 '16 at 14:31

hankchiutw

1,546
1
12
15

can i run it in the background because i dont want to have a dedicated console to this. i want something to make sure this node server is continuously running eventhough it may shutdown – Axil Mar 29 '16 at 14:54
yes, `pm2 start` starts the process in background, you can use `pm2 list`, `pm2 monit`, `pm2 logs` to check the process. If you want to monitor it remotely, you may need to use their [dashboard service](http://pm2.keymetrics.io/docs/usage/monitoring/). – hankchiutw Mar 29 '16 at 15:07
i like this so far. it works i think, so far tested for 12 hours still alive. i think this is it. but one thing is i did all your commands like pm2 list, pm2 monit,... I cant seem to find my process in the table list ? – Axil Mar 30 '16 at 03:35
then, what did you see? – hankchiutw Mar 30 '16 at 04:31
ok now i see it, maybe a console problem. http://imgur.com/Kpo9LyL. But no restart. If it dies does it automatically restart and show it in the table counter ? – Axil Mar 30 '16 at 05:27

score -1 · Answer 3 · answered Apr 03 '16 at 11:09

For the server to connect back, you might need to use some program (like forever) that checks the status of the server and restarts it when it dies. This, however, does not solve the problem for the clients. The clients also need to reinitialize the connection to the server when the server dies. Let me show you a minimal example of how you can make your clients connect back to the server.

The main idea is that you will listen for the "close" event from the client socket and will restart a connection if needed. Let us use OOP to better illustrate this.

function Connector(host, port)
{
  // Connection parameters
  this.host = host;
  this.port = port;

  this.socket = null;

  // Attempt to establish a connection with the server.
  this._connect();
}

Connector.prototype._connect = function()
{
  this.socket = net.connect(this.port, this.host);

  this.socket.on('connect', this._onConnect);
  this.socket.on('error', this._onError);
  this.socket.on('close', this._onClose);
  this.socket.on('data', this._onData);
}


Connector.prototype._onConnect = function()
{
  console.log('Incoming connection');
}

Connector.prototype._onError = function(err)
{
  console.error('Connection error: ' + err);
}

Connector.prototype._onClose = function(had_error)
{
  console.log('Reconnecting to server.');
  self._scheduleReconnection();
}


Connector.prototype._scheduleReconnection = function()
{
  self.socket.removeAllListeners();
  self.socket = null;

  // Try to connect to the server again after 2 seconds.
  setTimeout(self._connect.bind(self), 2000);
}

Connector.prototype._onData = function(data)
{
  console.log('Incoming data.')
}

With the Connector object you can now start a connection with:

connection = new Connector(127.0.0.1, 8080)

and you know this connection will constantly try to stay connected with the server.

Auto restart or Proper Handling socket.io server on exception in linux

3 Answers3