Send SIGINT to subprocess and also see output

Question

In a celery task, I am launching a subprocess and need (1) to be able to send a SIGINT while (2) also having access to the subprocess's stdout and stderr. I am able to do one or the other, but not both simultaneously.

I can send SIGINT when the command in subprocess is given as a list, or as a string prepended with bash:

proc = subprocess.Popen(,
    [sys.executable, "-m", path.to.module, myarg1, myarg2, ...],    # also works with f"/bin/bash -c {sys.executable} -m path.to.module {myarg1} {myarg2} ..."
    stdin=sys.stdin, stdout=PIPE, stderr=PIPE, shell=False
)

As far as I understand, both options are ultimately launching bash and it seems that only a running bash will react to SIGINT.

Conversely, running "python -m ..." means my program no longer reacts to the SIGINT, but on the other hand it allows me to start seeing the stdout/stderr and logging inside my python program:

proc = subprocess.Popen(,
    f"{sys.executable} -m path.to.module {myarg1} {myarg2} ..."
    stdin=sys.stdin, stdout=PIPE, stderr=PIPE, shell=False
)

With the above, now I'm no longer able to send SIGINT to my program but the logging is working.

How can I get both things to work at the same time? I've played around with shell=True and the various stdin/out/err tweaks, but no luck.

EDIT: With the top form (command as a list) and adding signal.signal() to my program in path.to.module I am able to both receive the SIGINT as well as see some output.

The first form does not run bash or any other shell, but it is much much better especially from a security perspective. In the second form the shell is probably swallowing the interrupt signal. When you say you need access to the subprocess's output, what code handles it, and what results do you see? — David Maze, Mar 13 '23 at 23:51
Setting the command as a list, or as a string prepended with "/bin/bash" (as commented on that line) means it's running bash no? And yes, the bottom form feels like the shell is swallowing the signal. I need to see the output of that `path.to.module` program, which is simply logging to stdout/err and file. No other code needs to handle it, I just need to see it. With the top form I can't see it, with the bottom form I can see it but I lose the ability to send `SIGINT`. — Jaime Salazar, Mar 14 '23 at 18:16
"Setting the command as a list" does not in and of itself cause a shell to be run, unless the path to the bash executable is the first element of that list. `sys.executable` is the Python executable, not the bash executable. — Charles Duffy, Mar 14 '23 at 21:26
I see now. I was basing it off a comment found on this post but I'm seeing that's only true when shell = True: https://stackoverflow.com/questions/10661457/why-does-shell-true-eat-my-subprocess-popen-stdout — Jaime Salazar, Mar 15 '23 at 12:24

J_H · Accepted Answer · 2023-03-14T00:02:36.237

need to be able to send a SIGINT ...

I disagree with the premise.

I believe the need is to "stop a running process" in some sensible way, and INT, USR1, HUP or other signals might suffice. The key is to have a signal handler active for the relevant signal.

The other detail is "access to ... stdout", which is entirely reasonable. That is, we do not want abrupt termination, but rather we want printf(3) buffers to be flushed by write(2) syscalls prior to termination.

So the crux is that we want child process (or maybe process group) to be sent a signal it is expecting, and then handle the signal nicely.

bash traps certain signals so that is definitely a concern.

proc = subprocess.Popen(
    ... , shell=False)

... both options are ultimately launching bash

No, at least not in this instance. The False argument means that bash is not involved here.

Consider defining a handler, and then doing

signal.signal(signal.SIGUSR1, handler)

The handler might choose to call sys.exit, in order to flush stdout buffers.

https://docs.python.org/3/library/signal.html#signal.signal

`path.to.module` is a while loop which breaks on SIGINT. Idc which signal it uses to break the loop, the issue is that it only detects the signal with the top form. But with that top form I stop seeing any of its output. Conversely, with the bottom form I can see the output but lose the ability to detect the SIGINT. Also, the command in the top form is a list or a string prepended with bash. I thought this means bash is launched regardless of shell=False. Finally, are you saying to add `signal.signal()` to `path.to.module`? Not sure how that would help since it's not detecting incoming signals — Jaime Salazar, Mar 14 '23 at 18:57
Please ignore my previous comment, it is now working with the top form but having added signal.signal() to `path.to.module`. Thanks! — Jaime Salazar, Mar 14 '23 at 20:25

Send SIGINT to subprocess and also see output

1 Answers1