2

What is best practices to prevent Node.js PRODUCTION server's crash due to an unknown and un-handled exception?

Ajay
  • 1,255
  • 1
  • 17
  • 30
  • The best practice is DOMAIN with Cluster. See [this](http://stackoverflow.com/questions/5999373/how-do-i-prevent-node-js-from-crashing-try-catch-doesnt-work) question – chrmcpn Apr 25 '17 at 17:28

1 Answers1

5

These days, the "fail fast" mentality is all the rage. You don't prevent it from crashing ... you let it crash and then have logic that both logs the error, informs you via email or text and then gracefully recovers. Here's how I do it:

Deploy using systemd (upstart would be ok too but is falling out of favor). Make sure in your .system file you have Restart logic. Deploy your node program as a Docker container. When it fails, the container will exit, systemd will notice it and restart. I also use logging-as-a-service so that I have all the logs in the could. Finally, we use an alerting mechanism (nagios, stackdriver, there are many others out there) so we know when something fails.

You asked for a best practice and that's what a lot of folks are doing now. But it's expensive. If you don't have any money or time or programmers to do this stuff, you'll need to compromise.

I wouldn't compromise on a few of these, though:

  1. Logging failures. Most SaaS logging companies have a free plan, so use it.

  2. Something to monitor your process and restart it. At the OS level Upstart and Systemd are fairly easy to configure. But if you can't, at least use node's "forever" package.

If you can't do anything like the above and truly have hit the bottom of the barrel, put this at the entrypoint to your node app:

process.on('uncaughtException', function(err) {
  //do something here
});

But for real, you want to fail and recover, learn from it, fix the bug and that will make your software stronger.

bbuckley123
  • 1,879
  • 13
  • 18
  • Thanks for the detailed information. I am using "forever" currently. Actually my Node.js server needs persistence connectivity with large number of users on TCP for sending push message realtime. Any restart of server disconnects all connections then all have to reconnect. Which is not desirable for my application. I did a lot of exception handling into the application but worried about any unknown and un-handle exception. The process.on('uncaughtException', function(err) .... are mentioned many places but not recommended to use. – Ajay Oct 02 '15 at 13:57
  • 1
    You could scale horizontally behind a load balancer. You will still lose connections on a fail-over, but you'd be more stable. AWS and GCE both have decent ones. HaProxy is good as well. Since you are doing messaging, you might want to leverage a fairly stable message broker such as 0MQ or even RabbitMQ, the latter would allow you to recover a "failed" message. Finally, re-initializing a TCP connection is usually the client's job. So you would want to program TCP-reconnection on the client-side app, not your server-side code. Code for failure, not perfection. – bbuckley123 Oct 02 '15 at 14:06
  • Fortunately I already did as you suggested, a HAProxy as load balancer in front of horizontal scalable Node.js servers and RabbitMQ for message delivery. Also Client is designed for re-connection at the event of any connectivity failure...Thanks for the more details....Should I use the "process.on('uncaughtException'..." to prevent server crash and log exception to repair later? – Ajay Oct 02 '15 at 14:26
  • 1
    Pragmatically, I would say no. Fail fast. But I know what it's like to have production crash and the freak-out scene it causes at work. You don't want to lose money or clients for the sake of obeying an in-vogue mantra. So yes. If it's prod and you are under the gun, use it. But use it wisely and log like crazy so you can fix the root issue instead of masking it. Good luck! – bbuckley123 Oct 02 '15 at 14:32
  • Great! My intension not to mask but log and correct on urgent basis. Thanks for the nice help. – Ajay Oct 02 '15 at 14:38
  • 1
    uncaughtException Method is not a good practice. See this question http://stackoverflow.com/questions/5999373/how-do-i-prevent-node-js-from-crashing-try-catch-doesnt-work – chrmcpn Apr 25 '17 at 17:26