45

Currently, I have two servers running on an EC2 instance (MongoDB and bottlepy). Everything works when I SSHed to the instance and started those two servers. However, when I closed the SSH session (the instance is still running), I lost those two servers. Is there a way to keep the server running after logging out? I am using Bitvise Tunnelier on Windows 7. The instance I am using is Ubuntu Server 12.04.3 LTS.

TTT
  • 4,354
  • 13
  • 73
  • 123
  • Possible duplicate of [In Linux, how to prevent a background process from being stopped after closing SSH client](https://stackoverflow.com/q/285015/608639), [How to make a program continue to run after log out from ssh?](https://stackoverflow.com/q/954302/608639), [Run a command in a shell and keep running the command when you close the session](https://stackoverflow.com/q/431521/608639), etc. – jww May 06 '19 at 21:47

6 Answers6

64

For those landing here from a google search, I would like to add tmux as another option. tmux is widely used for this purpose, and is preinstalled on new Ubuntu EC2 instances.

Managing a single session

Here is a great answer by Hamish Downer given to a similar question over at askubuntu.com:

I would use a terminal multiplexer - screen being the best known, and tmux being a more recent implementation of the idea. I use tmux, and would recommend you do to.

Basically tmux will run a terminal (or set of terminals) on a computer. If you run it on a remote server, you can disconnect from it without the terminal dying. Then when you login in again later you can reconnect, and see all the output you missed.

To start it the first time, just type

tmux

Then, when you want to disconnect, you do Ctrl+B, D (ie press Ctrl+B, then release both keys, and then press d)

When you login again, you can run

tmux attach

and you will reconnect to tmux and see all the output that happened. Note that if you accidentally lose the ssh connection (say your network goes down), tmux will still be running, though it may think it is still attached to a connection. You can tell tmux to detach from the last connection and attach to your new connection by running

tmux attach -d

In fact, you can use the -d option all the time. On servers, I have this in my >.bashrc

alias tt='tmux attach -d'

So when I login I can just type tt and reattach. You can go one step further >if you want and integrate the command into an alias for ssh. I run a mail client >inside tmux on a server, and I have a local alias:

alias maileo='ssh -t mail.example.org tmux attach -d'

This does ssh to the server and runs the command at the end - tmux attach -d The -t option ensures that a terminal is started - if a command is supplied then it is not run in a terminal by default. So now I can run maileo on a local command line and connect to the server, and the tmux session. When I disconnect from tmux, the ssh connection is also killed.

This shows how to use tmux for your specific use case, but tmux can do much more than this. This tmux tutorial will teach you a bit more, and there is plenty more out there.

Managing multiple sessions

This can be useful if you need to run several processes in the background simultaneously. To do this effectively, each session will be given a name.

Start (and connect to) a new named session:

tmux new-session -s session_name

Detach from any session as described above: Ctrl+B, D.

List all active sessions:

tmux list-sessions

Connect to a named session:

tmux attach-session -t session_name

To kill/stop a session, you have two options. One option is to enter the exit command while connected to the session you want to kill. Another option is by using the command:

tmux kill-session -t session_name
Joakim
  • 2,092
  • 1
  • 20
  • 23
  • 1
    when you want to disconnect/Detach, you press Ctrl+B then press D immediately! (ie press Ctrl+B, then release both keys, and then press d immediately) – Amar Kumar Dec 06 '20 at 16:14
16

If you don't want to run some process as a service (or via an apache module) you can (like I do for using IRC) use gnome-screen Install screen http://hostmar.co/software-small.

screen keeps running on your server even if you close the connection - and thus every process you started within will keep running too.

guntbert
  • 536
  • 6
  • 19
  • 3
    Screen is a good option, especially for OP's use case. A lot of people like tmux too, which is probably a better alternative for more advanced usage needs – adavea Nov 10 '16 at 16:31
  • how to install it on EC2 Ubuntu instance that I only have session – AMH Oct 25 '17 at 20:55
3

It would be nice if you provided more info about your environment but assuming it's Ubuntu Linux you can start the services in the background or as daemons.

sudo service mongodb start
nohup python yourbottlepyapp.py &

(Use nohup if you want are in a ssh session and want to prevent it from closing file descriptors)

You can also run your bottle.py app using Apache mod_wsgi. (Running under the apache service) More info here: http://bottlepy.org/docs/dev/deployment.html

Hope this helps.

Addition: (your process still runs after you exit the ssh session)

Take this example time.py

 import time
 time.sleep(3600)

Then run:

 $ python3 time.py &
 [1] 3027
 $ ps -Af | grep -v grep | grep  time.py
 ubuntu    3027  2986  0 18:50 pts/3    00:00:00 python3 time.py
 $ exit

Then ssh back to the server

 $ ps -Af | grep -v grep | grep  time.py
 ubuntu    3027     1  0 18:50 ?        00:00:00 python3 time.py

Process still running (notice with no tty)

Rico
  • 58,485
  • 12
  • 111
  • 141
  • I am running Ubuntu, and I will test your suggestion. Thanks! – TTT Jan 17 '14 at 19:47
  • I think the mongodb is running since `mongod` is included in the init.d folder. But the statement did not work 'python yourbottlepyapp.py &'... – TTT Jan 17 '14 at 20:44
  • What does it do ? the program exits? – Rico Jan 17 '14 at 20:45
  • Right. The python bottle server stopped once the SSH session is over. I guess maybe I should drop a .cong file in the /etc/init/ folder? – TTT Jan 17 '14 at 20:46
  • Best thing is probably to run it using apache mod_wsgi. – Rico Jan 17 '14 at 20:47
  • Running a process in the background with & does not detach it from the ssh session, and the python process will still be terminated when the ssh session is closed. Starting a service and running things in screen/tmux works differently, though I'm not sure why specifically they work differently – adavea Nov 10 '16 at 16:27
  • @adavea yes it does ! Check my addition. Don't downvote if you can't show examples. – Rico Nov 10 '16 at 18:53
  • Here is a php example: add(new DateInterval("PT1M")); while($d < $d2) { $d = new DateTime("now", $utc); file_put_contents("a", $d->format("H:i:s") . "\n", FILE_APPEND); sleep(1); } If you open 2 sessions, and run this file in the first, and tail -f "a" in the second, you will see that the file stops being written to as soon as you close your ssh session. It's the same with python, I just tested it. Some ssh clients use nohup with &, but most do not – adavea Nov 11 '16 at 14:32
  • @adavea ahh that is if you have a file descriptor open. It's more like 'Running a process in the background with & does not `necessarily` detach it from the ssh session'. You could have just suggested `nohup`. Check the edit... – Rico Nov 11 '16 at 16:27
  • I'm pretty sure it's not related to the file descriptor. To double check, I altered my php code to make a GET request to a web service I have instead of writing to a file, and it still terminates once I close my ssh session. I think you might just be using an ssh client that runs background tasks in a nohup automatically? That or python has a special way of running things that causes them not to be closed when you terminate your ssh connection – adavea Nov 14 '16 at 14:39
  • Running standard ssh session using mac ssh client to a centos box. Nothing out of the ordinary. What are you running ? – Rico Nov 14 '16 at 15:18
2

You will want the started services to disconnect from the controlling terminal. I would suggest you use nohup to do that, e.g.

ssh my.server "/bin/sh -c nohup /path/to/service"

you may need to put an & in there (in the quotes) to run it in the background.

As others have commented, if you run proper init scripts to start/stop services (or ubuntu's service command), you should not see this issue.

abligh
  • 24,573
  • 4
  • 47
  • 84
2

You can use pm2 to keep your Node.js application running on an EC2 instance after disconnecting from SSH.

PM2 is a process manager for Node.js applications that provides built-in support for keeping applications running in the background and automatically restarting them if they crash. You can start your Node.js application with PM2 and it will continue running even after you disconnect from the EC2 instance.

Here are the basic steps to use PM2:

Install PM2: npm install pm2 -g

Start your Node.js application with PM2: pm2 start <your-app.js>

ubuntu@ip-172-31-4-234:~$ pm2 start index.js
[PM2] Applying action restartProcessId on app [index](ids: [ 0 ])
[PM2] [index](0) ✓
[PM2] Process successfully started
┌────┬──────────┬─────────────┬─────────┬─────────┬──────────┬────────┬──────┬───────────┬──────────┬──────────┬──────────┬──────────┐
│ id │ name     │ namespace   │ version │ mode    │ pid      │ uptime │ ↺    │ status    │ cpu      │ mem      │ user     │ watching │
├────┼──────────┼─────────────┼─────────┼─────────┼──────────┼────────┼──────┼───────────┼──────────┼──────────┼──────────┼──────────┤
│ 0  │ index    │ default     │ N/A     │ fork    │ 4559     │ 0s     │ 16   │ online    │ 0%       │ 7.2mb    │ ubuntu   │ disabled │

Verify that your application is running: pm2 list

ubuntu@ip-172-31-4-234:~$ pm2 list
┌────┬──────────┬─────────────┬─────────┬─────────┬──────────┬────────┬──────┬───────────┬──────────┬──────────┬──────────┬──────────┐
│ id │ name     │ namespace   │ version │ mode    │ pid      │ uptime │ ↺    │ status    │ cpu      │ mem      │ user     │ watching │
├────┼──────────┼─────────────┼─────────┼─────────┼──────────┼────────┼──────┼───────────┼──────────┼──────────┼──────────┼──────────┤
│ 0  │ index    │ default     │ N/A     │ fork    │ 4040     │ 110s   │ 15   │ online    │ 0%       │ 58.5mb   │ ubuntu   │ disabled │
└────┴──────────┴─────────────┴─────────┴─────────┴──────────┴────────┴──────┴────────���──┴──────────┴──────────┴──────────┴──────────┘

To stop your application: pm2 stop <app-name-or-id>

ubuntu@ip-172-31-4-234:~$ pm2 stop index
[PM2] Applying action stopProcessId on app [index](ids: [ 0 ])
[PM2] [index](0) ✓
┌────┬──────────┬─────────────┬─────────┬─────────┬──────────┬────────┬──────┬───────────┬──────────┬──────────┬──────────┬──────────┐
│ id │ name     │ namespace   │ version │ mode    │ pid      │ uptime │ ↺    │ status    │ cpu      │ mem      │ user     │ watching │
├────┼──────────┼─────────────┼─────────┼─────────┼──────────┼────────┼──────┼───────────┼──────────┼──────────┼──────────┼──────────┤
│ 0  │ index    │ default     │ N/A     │ fork    │ 0        │ 0      │ 15   │ stopped   │ 0%       │ 0b       │ ubuntu   │ disabled │

To restart your application: pm2 restart <app-name-or-id>

ubuntu@ip-172-31-4-234:~$ pm2 restart index
Use --update-env to update environment variables
[PM2] Applying action restartProcessId on app [index](ids: [ 0 ])
[PM2] [index](0) ✓
┌────┬──────────┬─────────────┬─────────┬─────────┬──────────┬────────┬──────┬───────────┬──────────┬──────────┬──────────┬──────────┐
│ id │ name     │ namespace   │ version │ mode    │ pid      │ uptime │ ↺    │ status    │ cpu      │ mem      │ user     │ watching │
├────┼──────────┼─────────────┼─────────┼─────────┼──────────┼────────┼──────┼───────────┼──────────┼──────────┼──────────┼──────────┤
│ 0  │ index    │ default     │ N/A     │ fork    │ 4528     │ 0s     │ 15   │ online    │ 0%       │ 15.4mb   │ ubuntu   │ disabled │
└────┴──────────┴─────────────┴─────────┴─────────┴──────────┴────────┴──────┴────────���──┴──────────┴──────────┴──────────┴──────────┘

To view logs for your application: pm2 logs <app-name-or-id>

ubuntu@ip-172-31-4-234:~$ pm2 logs index
[TAILING] Tailing last 15 lines for [index] process (change the value with --lines option)
/home/ubuntu/.pm2/logs/index-error.log last 15 lines:
0|index    |     at listenInCluster (node:net:1788:12)
0|index    |     at Server.listen (node:net:1876:7)
0|index    |     at Function.listen (webpack://zen-technical-interview-1/./node_modules/express/lib/application.js?:635:24)
0|index    |     at eval (webpack://zen-technical-interview-1/./src/index.ts?:70:5)
0|index    |     at ./src/index.ts (/home/ubuntu/index.js:1315:1)
0|index    |     at __webpack_require__ (/home/ubuntu/index.js:1690:42)
0|index    |     at /home/ubuntu/index.js:1719:37
0|index    |     at Object.<anonymous> (/home/ubuntu/index.js:1721:12)
0|index    |     at Module._compile (node:internal/modules/cjs/loader:1254:14) {
0|index    |   code: 'EADDRINUSE',
0|index    |   errno: -98,
0|index    |   syscall: 'listen',
0|index    |   address: '::',
0|index    |   port: 3000
0|index    | }

/home/ubuntu/.pm2/logs/index-out.log last 15 lines:
0|index    | --- [
0|index    |   { scenario: 1, provider: { id: 2, name: 'Provider 2' }, price: 250 },
0|index    |   { scenario: 2, provider: { id: 1, name: 'Provider 1' }, price: 300 },
0|index    |   { scenario: 3, provider: { id: 3, name: 'Provider 3' }, price: 200 },

To start PM2 automatically when the server starts, you can use the pm2 startup command, which will generate a startup script and add it to the system startup configuration.

By using PM2, you can ensure that your Node.js application remains running in the background even if you disconnect from your EC2 instance, and can easily manage your application with PM2's built-in commands.

s00103898-276165-15433
  • 750
  • 8
  • 11
0

If on Linux based instances putting the job in the background followed by disown seems to do the job.

$ ./script & $ disown

sridharn
  • 111
  • 7