Work around ssh does not forward signal

Question

How can I work around the problem that ssh does not forward the SIGTERM signal?

I want print-signal.py to terminate of the ssh root@localhost process terminates:

ssh root@localhost /root/print-signal.py

Unfortunately only the ssh process itself gets the signal, not the remote command (print-signal.py). The remote command does not terminate :-(

Since openssh does not forward the SIGTERM to the remote command, I am searching for a work-around.

How to terminate print-signal.py if ssh root@localhost ... terminates?

This is a follow-up question to: Forwarding SIGTERM over ssh

It seems impossible to send over a particular signal and kill a command with that particular signal. `ssh -t server cmd` allows a clean termination of `cmd` on `server`, but this is due to `SIGHUP`. if you use `ssh server cmd`, and kill the ssh, `cmd` will become a child of `init` and will be eventually killed off by `SIGPIPE`. The only way to send `SIGTERM` to the process is possible by a secondary ssh (`ssh server "pgrep -U $USER cmd | xargs kill -SIGTERM"`). But again, this is problematic if you have multiple instances of such ssh running. — kvantour, Feb 05 '18 at 13:45

kvantour · Accepted Answer · 2018-02-03T23:09:56.533

Disclaimer: the answer below is not for SIGTERM, but SIGINT. This is not an answer to the question due to oversight.

The problem you are observing is due to a missing tty which should be in control of the process you try to run. If there is no tty available, ssh is unable to send the signals to the process. When you use the option -t to the ssh command, it will force pseudo-terminal allocation which makes it possible to send signals over ssh :

ssh -t root@localhost /root/print-signal.py

man ssh -t Force pseudo-terminal allocation. This can be used to execute arbitrary screen-based programs on a remote machine, which can be very useful, e.g. when implementing menu services. Multiple -t options force tty allocation, even if ssh has no local tty.

A very nice explanation how and why is given by Giles on unix.stackexchange.

Here you see how it works :

[terminal 1]% ssh server ./print_signal.py

On an other terminal you see then that print_signal.py is running on PID=26992 under the ssh with PID=26991 without a tty (username@notty)

[terminal 2]% ssh server ps -f                                                                                                                                                                                                            
UID        PID  PPID  C STIME TTY          TIME CMD
username 26991 26989  0 17:06 ?        00:00:00 sshd: username@notty
username 26992 26991  0 17:06 ?        00:00:00 python ./print_signal.py
username 27347 27345  0 17:07 ?        00:00:00 sshd: username@notty
username 27348 27347  0 17:07 ?        00:00:00 ps -f

After killing the ssh process with kill or CTRL-C, the process is still active but runs now under /sbin/init (PPID=1)

[terminal 2]% ssh server ps -f
UID        PID  PPID  C STIME TTY          TIME CMD
username 26992     1  0 17:06 ?        00:00:00 python ./print_signal.py
username 27453 27451  0 17:08 ?        00:00:00 sshd: username@notty
username 27454 27453  5 17:08 ?        00:00:00 ps -f

using the -t flag nicely kills the process on the other side:

[terminal 1]% ssh -t server ./print_signal.py

On an other terminal you see then that print_signal.py is running on PID=39277 under the ssh with PID=39276 bound to a tty (username@pts/10)

[terminal 2]% ssh server ps -U username -f
UID        PID  PPID  C STIME TTY          TIME CMD
username 39276 39274  0 17:22 ?        00:00:00 sshd: username@pts/10
username 39277 39276  1 17:22 pts/10   00:00:00 python ./print_signal.py
username 39317 39314  0 17:22 ?        00:00:00 sshd: username@notty
username 39318 39317  5 17:22 ?        00:00:00 ps -U username -f

After killing the ssh process

[terminal 1]% ssh -t server ./print_signal.py
My PID: 39277
^CCaught signal SIGINT (2), exiting.
Connection to server closed

The process is now clearly terminated on the other server

[terminal 2]% ssh server ps -f
UID        PID  PPID  C STIME TTY          TIME CMD
username 39768 39765  0 17:26 ?        00:00:00 sshd: username@notty
username 39769 39768  6 17:26 ?        00:00:00 ps -U username -f

@guettli also works on `openssh-5.3` and on unix systems with `Sun_SSH_1.1.6, SSH protocols 1.5/2.0, OpenSSL 0x0090704f` — kvantour, Jan 30 '18 at 10:01
@guettli, did you try multiple combinations of `-t`, eg. `ssh -t -t localhost cmd`. This might be helpful, multiple combinations force the tty to exist, even if the cmd is not executed in a tty. — kvantour, Jan 31 '18 at 08:59
The question asks about SIGTERM. ^C triggers SIGINT, not SIGTERM. There are other signals that you can trigger through a TTY, but SIGTERM isn't one of them. — Kenster, Feb 02 '18 at 16:41

score 2 · Answer 2 · edited Jan 18 '21 at 12:13

2

I have just been having exactly this problem. While I haven't figured out the exact cause, what happens when ssh is terminated is that your process is re-parented to init. You can tell your process to ask for a signal when it's parent dies instead using prctl.

If you use python-prctl, put the following early on in /root/print-signal.py

import signal
import prctl

prctl.set_pdeathsig(signal.SIGTERM)

edited Jan 18 '21 at 12:13

famzah

1,462
18
21

answered Dec 05 '18 at 16:04

John Oxley

14,698
18
53
78

Many thanks, it works for me just fine. I had the same problem with a flask server started over ssh : the flask server nerver ends whatever you do. – Michael Dussere Apr 09 '21 at 13:55

score 0 · Answer 3 · answered Jan 24 '18 at 10:14

You could write a wrapper which terminates if the parent process is the init process:

ssh root@localhost terminate-command-if-parent-is-lost print-signal.py

The tool terminate-command-if-parent-is-lost needs to do this:

Start the argv as subprocess (in this example print-signal.py). Then it checks every second the status of its parent pid (in Python os.getppid()).

If ppid is 1 (the init process), then the print-signal.py process has lost its parent. This means "ssh root@localhost ..." was terminated (or connection was closed).

Now terminate-command-if-parent-is-lost terminates the subprocess.

Feels too complicated. I guess there is an easier solution. – guettli Jan 24 '18 at 10:15 — guettli, Jan 24 '18 at 10:15

score 0 · Answer 4 · edited Feb 05 '18 at 10:26

0

The shell can easily help you for such problem, if it supports some builtin variable( $! ). Here is a basic shell solution by replacing your command by a very long sleep.

ssh server '(sleep 100000 & (MID=$!; A=n; while [ "$A" != y ];do echo "i am process $$ want kill "$MID" y/n?"; read A; done; kill -TERM $MID))'

The script sent in the command is running on the remote host.

edited Feb 05 '18 at 10:26

Maytham Fahmi

31,138
14
118
137

answered Feb 02 '18 at 14:26

Stéphane Le Men

137
3

Work around ssh does not forward signal

4 Answers4

Linked