24

I am using forever js to keep my node server running 24/7 on AWS EC2.

I use this command

forever start index.js

However, I notice that some time it randomly stops the process and my site goes down. I have to manually ssh into my server to run it again by doing:

forever restartall

And then it goes backup. Is there any way by which I can define a timeout, lets say if the server/website does not respond for 200 in 5 sec, then restart all forever process automatically?

I am new to this, if any one can give me step by step example for my case, it would be awesome.

Faizan
  • 1,847
  • 8
  • 40
  • 63
  • 2
    If this is a *NIX machine, then you could simply define a shell script that runs on a cron job to check the state of the process and restart if necessary. – Evan Bechtol May 12 '16 at 20:50
  • Possible duplicate of [Automatically start forever (node) on system restart](http://stackoverflow.com/questions/13385029/automatically-start-forever-node-on-system-restart) – Mark B May 13 '16 at 14:44
  • @EvanBechtol Its on AWS EC2 t2.micro linux ubuntu.. I am totally new to this, can you please give the code example how to and where to define and run the shell script? Thanks, i'll mark it best answer :) – Faizan May 13 '16 at 17:26
  • 1
    @Fizan sure, I'll post an example today. Sorry for the delay – Evan Bechtol May 14 '16 at 21:04
  • Check the already existing answer http://stackoverflow.com/questions/13385029/automatically-start-forever-node-on-system-restart – Chinni Jun 02 '16 at 19:37

6 Answers6

34

A (NodeJS) server should not stop for no reason. Most of the time, it's because of a 500 Error that have not been catched and stop the server, then you will have to restart it. forever is using node by default to start your server.

nodemon is a npm package that restart your server when the code changes or when your server stops.

You can use forever and nodemon together by doing :

forever start nodemon --exitcrash app.js

or

forever start -c nodemon app.js

Or, as suggested in other answers, you can use PM2, which would be better for production !

boehm_s
  • 5,254
  • 4
  • 30
  • 44
28

I'd suggest the usage of PM2

This is the best option to run on a production server.

What are the advantages of running your application this way?

  • It's easy to setup and run.
  • PM2 will automatically restart your application if it crashes.
  • PM2 will keep a log of your unhandled exceptions - in this case, in a file at /home/safeuser/.pm2/logs/app-err.log.
  • With one command, PM2 can ensure that any applications it manages restart when the server reboots. Basically meaning your node application will start as a service.

ref: https://www.digitalocean.com/community/tutorials/how-to-use-pm2-to-setup-a-node-js-production-environment-on-an-ubuntu-vps

Community
  • 1
  • 1
Vishnu Mishra
  • 3,683
  • 2
  • 25
  • 36
3

So this is an example of using cronto run scripts that can restart service/perform some automated task. Basically, I created some scripts that I need to run at certain time intervals on my server. For your case, you want to make a script that will automatically check the state of your forever.js and if it returns a bad response, run the forever restartall command that you mention above. crontab

You can set this up by creating a new crontab entry on the server. As far as the script goes, I'm by no means a bash script guru; I made a simple script that works for me. Here is an example of checking a service on my machine, restarting it if it is not running.

#!/bin/bash
zabbix_server="service zabbix-server"
zabbix_agent="service zabbix-agent"
logfile=zabbix_auto_restart.log
logfilePath=/etc/scripts/zabbix/$logfile
zabbix_server_running=0
zabbix_agent_running=0

grep_agent (){
        local retval=$(ps -ef | grep -v grep | grep zabbix_agentd | wc -l)
        echo $retval
}

grep_server (){
        local retval=$(ps -ef | grep -v grep | grep zabbix_server | wc -l)
        echo $retval
}

check_zabbix_agentd (){
        if (( $(grep_agent) <= 0 ))
        then
           sudo /etc/init.d/zabbix-agent start
           echo `date` "$zabbix_agent was stopped... Restarting" >> $logfilePath
           echo "************************************************" >> $logfilePath

           #Send email to notify that the script ran
           echo "$(date) $zabbix_agent was restarted from zabbix_restart.sh" | mutt -s "Zabbix Auto-restart Script Just Ran" <my-email>

        else
           let zabbix_agent_running=1
        fi
}

check_zabbix_server (){
        if (( $(grep_server) <= 0 ))
        then
           sudo /etc/init.d/zabbix-server start
           echo `date` "$zabbix_server was stopped... Restarting" >> $logfilePath
           echo "************************************************" >> $logfilePath

           #Send email to notify that the script ran
           echo "$(date) $zabbix_server was restarted from zabbix_restart.sh" | mutt -s "Zabbix Auto-restart Script Just Ran" evan.bechtol@ericsson.com

        else
           let zabbix_server_running=1
        fi
}

main_loop (){
        until ((zabbix_server_running == 1 && zabbix_agent_running == 1));
        do
                check_zabbix_agentd
                check_zabbix_server
                sleep 1.5
        done
}

main_loop
Evan Bechtol
  • 2,855
  • 2
  • 18
  • 36
2

This video is part of an awesome node-deployment-series ending up with using a service on a UNIX-server... Maybe it helps you. By the way, it's worth watching all four videos... Youtube-Link

Argee
  • 1,216
  • 1
  • 12
  • 22
1

Have you ever considered pm2 as an alternative. pm2 has some pretty neat features like:

  • Running a cluster
  • Restarting instances in the cluster one by one (zero downtime deployments)
  • Resource monitoring (shows realtime cpu and memory usage)
  • Manage all pm2 processes using the cli

You can control restarts also by adding a --restart-delay <delay> option.

Hereby a full output of the help of this tool.

pm2 --help

                    -------------
Looking for a complete monitoring and management tool for PM2?
 _                             _        _            _
| | _____ _   _ _ __ ___   ___| |_ _ __(_) ___ ___  (_) ___
| |/ / _ \ | | | '_ ` _ \ / _ \ __| '__| |/ __/ __| | |/ _ \
|   <  __/ |_| | | | | | |  __/ |_| |  | | (__\__ \_| | (_) |
|_|\_\___|\__, |_| |_| |_|\___|\__|_|  |_|\___|___(_)_|\___/
     |___/

                      Features

               - Real Time Dashboard
               - CPU/Memory monitoring
               - HTTP monitoring
               - Event notification
               - Custom value monitoring
               - Real Time log display

                      Checkout

               https://keymetrics.io/

                    -------------

[PM2] Spawning PM2 daemon
[PM2] PM2 Successfully daemonized

Usage: pm2 [cmd] app

Commands:

start [options] <file|json|stdin|app_name|pm_id...>                  start and daemonize an app
deploy <file|environment>                                            deploy your json
startOrRestart <json>                                                start or restart JSON file
startOrReload <json>                                                 start or gracefully reload JSON file
startOrGracefulReload <json>                                         start or gracefully reload JSON file
stop [options] <id|name|all|json|stdin...>                           stop a process (to start it again, do pm2 restart <app>)
restart [options] <id|name|all|json|stdin...>                        restart a process
scale <app_name> <number>                                            scale up/down a process in cluster mode depending on total_number param
reload <name|all>                                                    reload processes (note that its for app using HTTP/HTTPS)
gracefulReload <name|all>                                            gracefully reload a process. Send a "shutdown" message to close all connections.
id <name>                                                            get process id by name
delete <name|id|script|all|json|stdin...>                            stop and delete a process from pm2 process list
sendSignal <signal> <pm2_id|name>                                    send a system signal to the target process
ping                                                                 ping pm2 daemon - if not up it will launch it
updatePM2                                                            update in-memory PM2 with local PM2
update                                                               (alias) update in-memory PM2 with local PM2
install|module:install <module|git:/>                                install or update a module and run it forever
module:update <module|git:/>                                         update a module and run it forever
module:generate [app_name]                                           Generate a sample module in current folder
uninstall|module:uninstall <module>                                  stop and uninstall a module
publish|module:publish                                               Publish the module you are currently on
set <key> <value>                                                    sets the specified config <key> <value>
multiset <value>                                                     multiset eg "key1 val1 key2 val2
get [key]                                                            get value for <key>
conf [key] [value]                                                   get / set module config values
config <key> [value]                                                 get / set module config values
unset <key>                                                          clears the specified config <key>
interact [options] [secret_key|command] [public_key] [machine_name]  linking action to keymetrics.io - command can be stop|info|delete|restart
link [options] [secret_key|command] [public_key] [machine_name]      linking action to keymetrics.io - command can be stop|info|delete|restart
web                                                                  launch a health API on port 9615
dump                                                                 dump all processes for resurrecting them later
save                                                                 (alias) dump all processes for resurrecting them later
resurrect                                                            resurrect previously dumped processes
startup [platform]                                                   auto resurrect process at startup. [platform] = ubuntu, centos, redhat, gentoo, systemd, darwin, amazon
logrotate                                                            copy default logrotate configuration
generate                                                             generate an ecosystem.json configuration file
ecosystem                                                            generate an ecosystem.json configuration file
reset <name|id|all>                                                  reset counters for process
describe <id>                                                        describe all parameters of a process id
desc <id>                                                            (alias) describe all parameters of a process id
info <id>                                                            (alias) describe all parameters of a process id
show <id>                                                            (alias) describe all parameters of a process id
list                                                                 list all processes
ls                                                                   (alias) list all processes
l                                                                    (alias) list all processes
status                                                               (alias) list all processes
jlist                                                                list all processes in JSON format
prettylist                                                           print json in a prettified JSON
monit                                                                launch termcaps monitoring
m                                                                    (alias) launch termcaps monitoring
flush                                                                flush logs
reloadLogs                                                           reload all logs
logs [options] [id|name]                                             stream logs file. Default stream all logs
kill                                                                 kill daemon
pull <name> [commit_id]                                              updates repository for a given app
forward <name>                                                       updates repository to the next commit for a given app
backward <name>                                                      downgrades repository to the previous commit for a given app
gc                                                                   force PM2 to trigger garbage collection
deepUpdate                                                           performs a deep update of PM2
*

Options:

-h, --help                           output usage information
-V, --version                        output the version number
-v --version                         get version
-s --silent                          hide all messages
-m --mini-list                       display a compacted list without formatting
-f --force                           force actions
-n --name <name>                     set a <name> for script
-i --instances <number>              launch [number] instances (for networked app)(load balanced)
-l --log [path]                      specify entire log file (error and out are both included)
-o --output <path>                   specify out log file
-e --error <path>                    specify error log file
-p --pid <pid>                       specify pid file
-k --kill-timeout <delay>            delay before sending final SIGKILL signal to process
--max-memory-restart <memory>        specify max memory amount used to autorestart (in megaoctets)
--restart-delay <delay>              specify a delay between restarts (in milliseconds)
--env <environment_name>             specify environment to get specific env variables (for JSON declaration)
-x --execute-command                 execute a program using fork system
-u --user <username>                 define user when generating startup script
--hp <home path>                     define home path when generating startup script
-c --cron <cron_pattern>             restart a running process based on a cron pattern
-w --write                           write configuration in local folder
--interpreter <interpreter>          the interpreter pm2 should use for executing app (bash, python...)
--interpreter-args <arguments>       interpret options (alias of --node-args)
--log-date-format <momentjs format>  add custom prefix timestamp to logs
--no-daemon                          run pm2 daemon in the foreground if it doesn't exist already
--skip-env                           do not refresh environmnent on restart/reload
--source-map-support                 force source map support
--only <application-name>            with json declaration, allow to only act on one application
--disable-source-map-support         force source map support
--merge-logs                         merge logs from different instances but keep error and out separated
--watch [paths]                      watch application folder for changes
--ignore-watch <folders|files>       folder/files to be ignored watching, chould be a specific name or regex - e.g. --ignore-watch="test node_modules "some scripts""
--node-args <node_args>              space delimited arguments to pass to node in cluster mode - e.g. --node-args="--debug=7001 --trace-deprecation"
--no-color                           skip colors
--no-vizion                          start an app without vizion feature (versioning control)
--no-autorestart                     start an app without automatic restart
--no-treekill                        Only kill the main process, not detached children
--no-pmx                             start an app without pmx
--no-automation                      start an app without pmx

Basic Examples:

Start an app using all CPUs available + set a name :
$ pm2 start app.js -i 0 --name "api"

Restart the previous app launched, by name :
$ pm2 restart api

Stop the app :
$ pm2 stop api

Restart the app that is stopped :
$ pm2 restart api

Remove the app from the process list :
$ pm2 delete api

Kill daemon pm2 :
$ pm2 kill

Update pm2 :
$ npm install pm2@latest -g ; pm2 update

More examples in https://github.com/Unitech/pm2#usagefeatures

Deployment help:

$ pm2 deploy help
Marco
  • 4,817
  • 5
  • 34
  • 75
  • 1
    While investigating this matter at some point earlier I read about several problems with forever. PM2 was my choice based on that and I don't have any reason to complain despite months of usage. – Carlos Jimenez Bermudez Jun 08 '16 at 21:55
0

Figured it out. The problem was due to less RAM available in my EC2 instance. I upgraded RAM to 2 GB and have not experienced the crash so far.

Node can also make uncaught exception errors but my major problem was due to less memory.

Hope this helps others.

Faizan
  • 1,847
  • 8
  • 40
  • 63
  • 6
    While this was the answer to your problem, it does not really the answer the question posed. You may just come across this issue again in the future with more load and scaling vertically usually isn't the answer. – juz Jun 06 '16 at 16:37
  • 2
    You should really check out pm2 as in my answer above. You can see live memory usage at any point in time using `pm2 monit ` – Marco Jun 08 '16 at 11:10
  • Also check pm2 option to prevent memory leak issues by auto restartin a process if certain memory is used by the process. `--max-memory-restart` – Marco Apr 08 '19 at 12:44