Unable to pkill a subprocess

Question

Consider:

import subprocess
import time

def run_instance():
    command = "sleep 30 >out 2>err &"
    subprocess.check_output(command, shell=True, start_new_session=True)

def kill_instance():
    print(subprocess.check_output(f"pgrep -f 30", shell=True))
    subprocess.check_output(f"pkill -f 30", shell=True)

run_instance()
time.sleep(1)
kill_instance()

The output (run under Ubuntu) is:

b'17483\n17484\n'
Traceback (most recent call last):
  File "test.py", line 15, in <module>
    kill_instance()
  File "test.py", line 10, in kill_instance
    subprocess.check_output(f"pkill -f 30", shell=True)
  File "/usr/lib/python3.8/subprocess.py", line 411, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.8/subprocess.py", line 512, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command 'pkill -f 30' died with <Signals.SIGTERM: 15>.

Why does pkill fail even though pgrep finds the process?

I suppose you're trying to kill too much processes and don't have enough permissions. On my machine `pgrep -f 30` returns 17 processes now. — STerliakov, Jul 24 '22 at 11:36
Try to add a `-a` to the `pgrep` to see the process names. Likely a permission issue like @SUTerliakov said — dev93, Jul 24 '22 at 11:39
@SUTerliakov No, it's only two processes (one of which is `pgrep` itself), just as it appears on the output in the question. I added `-a` and got: `b'17490 sleep 30\n17491 /bin/sh -c pgrep -af 30\n'` — AlwaysLearning, Jul 24 '22 at 11:42
@dev93 Why would there be a permission issue when it's the same script that created these processes? (see my reply to @SUTerliakov) — AlwaysLearning, Jul 24 '22 at 12:11
Well, there you have your answer, your python code is succesfully sending a SIGTERM signal to your sleep ... and to itself. and that's the "error" you got. — dev93, Jul 24 '22 at 12:23
@dev93 Not itself, but `pgrep`... In fact, if I change `pkill -f 30` for `pkill sleep`, I get the same error. — AlwaysLearning, Jul 24 '22 at 12:28
Incidentally, why use `pkill` when you have the process handle and launched it yourself? Or is this just a demo? — 2e0byo, Jul 24 '22 at 13:44
Rewrite `run_instance` to `return subprocess.Popen(['sleep', '30'], stdout=open('out', 'w'), stderr=open('err', 'w'))` and you'll no longer be creating unnecessary problems for yourself that you need `pkill` to solve; instead you can just call `kill()` on that returned object. — Charles Duffy, Jul 24 '22 at 13:50
It's possible to make it work with `shell=True` also -- take out the `&`, use the shell builtin `exec`, and switch to `subprocess.Popen` — Charles Duffy, Jul 24 '22 at 13:55

STerliakov · Answer 1 · 2022-07-24T13:59:55.833

Common way

The most clean, IMO, way here is to just manage your instance within python (note you don't even need shell=True):

import subprocess
import time


def start_instance():
    with open('err', 'w') as err, open('out', 'w') as out:
        return subprocess.Popen(['/usr/bin/sleep', '30'], stdout=out, stderr=err)

def kill_instance(proc):
    proc.kill()

process = start_instance()
time.sleep(1)
kill_instance(process)

Your code

Disclaimer: you shouldn't be doing this in production.

You are killing your pkill. Here's the same with more verbose output:

import subprocess
import time

def run_instance():
    command = "sleep 30 >out 2>err &"
    subprocess.check_output(command, shell=True, start_new_session=True)

def kill_instance():
    print(subprocess.check_output(f"pgrep -f 30 -la", shell=True, universal_newlines=True))  # More verbose output
    subprocess.check_output(f"pkill -f 30", shell=True)

run_instance()
time.sleep(1)
kill_instance()

This outputs on my machine:

2446344 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 5434 -container-ip 172.30.0.2 -container-port 5434
2446350 /usr/bin/docker-proxy -proto tcp -host-ip :: -host-port 5434 -container-ip 172.30.0.2 -container-port 5434
2446618 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 8080 -container-ip 172.30.0.4 -container-port 8080
2446624 /usr/bin/docker-proxy -proto tcp -host-ip :: -host-port 8080 -container-ip 172.30.0.4 -container-port 8080
2446657 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 5001 -container-ip 172.30.0.5 -container-port 5000
2446666 /usr/bin/docker-proxy -proto tcp -host-ip :: -host-port 5001 -container-ip 172.30.0.5 -container-port 5000
2470568 kworker/u8:30-i915
2484058 /usr/lib/firefox/firefox -contentproc -childID 2176 -isForBrowser -prefsLen 41465 -prefMapSize 226778 -jsInitLen 277276 -parentBuildID 20220623063721 -appDir /usr/lib/firefox/browser 7462 true tab
2484060 /usr/lib/firefox/firefox -contentproc -childID 2177 -isForBrowser -prefsLen 41465 -prefMapSize 226778 -jsInitLen 277276 -parentBuildID 20220623063721 -appDir /usr/lib/firefox/browser 7462 true tab
2484067 /usr/lib/firefox/firefox -contentproc -childID 2178 -isForBrowser -prefsLen 41465 -prefMapSize 226778 -jsInitLen 277276 -parentBuildID 20220623063721 -appDir /usr/lib/firefox/browser 7462 true tab
2484121 /usr/lib/firefox/firefox -contentproc -childID 2179 -isForBrowser -prefsLen 41465 -prefMapSize 226778 -jsInitLen 277276 -parentBuildID 20220623063721 -appDir /usr/lib/firefox/browser 7462 true tab
2484283 /usr/lib/firefox/firefox -contentproc -childID 2181 -isForBrowser -prefsLen 41465 -prefMapSize 226778 -jsInitLen 277276 -parentBuildID 20220623063721 -appDir /usr/lib/firefox/browser 7462 true tab
2484293 /usr/lib/firefox/firefox -contentproc -childID 2182 -isForBrowser -prefsLen 41465 -prefMapSize 226778 -jsInitLen 277276 -parentBuildID 20220623063721 -appDir /usr/lib/firefox/browser 7462 true tab
2484392 /usr/lib/firefox/firefox -contentproc -childID 2183 -isForBrowser -prefsLen 41465 -prefMapSize 226778 -jsInitLen 277276 -parentBuildID 20220623063721 -appDir /usr/lib/firefox/browser 7462 true tab
2484394 /usr/lib/firefox/firefox -contentproc -childID 2184 -isForBrowser -prefsLen 41465 -prefMapSize 226778 -jsInitLen 277276 -parentBuildID 20220623063721 -appDir /usr/lib/firefox/browser 7462 true tab
2484423 /usr/lib/firefox/firefox -contentproc -childID 2185 -isForBrowser -prefsLen 41465 -prefMapSize 226778 -jsInitLen 277276 -parentBuildID 20220623063721 -appDir /usr/lib/firefox/browser 7462 true tab
2484442 /usr/lib/firefox/firefox -contentproc -parentBuildID 20220623063721 -prefsLen 41465 -prefMapSize 226778 -appDir /usr/lib/firefox/browser 7462 true socket
2484491 /usr/lib/firefox/firefox -contentproc -childID 2186 -isForBrowser -prefsLen 41465 -prefMapSize 226778 -jsInitLen 277276 -parentBuildID 20220623063721 -appDir /usr/lib/firefox/browser 7462 true tab
2484546 /usr/lib/firefox/firefox -contentproc -childID 2187 -isForBrowser -prefsLen 41465 -prefMapSize 226778 -jsInitLen 277276 -parentBuildID 20220623063721 -appDir /usr/lib/firefox/browser 7462 true tab
2487379 sleep 30
2487381 /bin/sh -c pgrep -f 30 -la

pkill: killing pid 2446344 failed: Operation not permitted
pkill: killing pid 2446350 failed: Operation not permitted
pkill: killing pid 2446618 failed: Operation not permitted
pkill: killing pid 2446624 failed: Operation not permitted
pkill: killing pid 2446657 failed: Operation not permitted
pkill: killing pid 2446666 failed: Operation not permitted
pkill: killing pid 2470568 failed: Operation not permitted
Traceback (most recent call last):
  File "/tmp/a.py", line 14, in <module>
    kill_instance()
  File "/tmp/a.py", line 10, in kill_instance
    subprocess.check_output(f"pkill -f 30", shell=True)
  File "/usr/local/lib/python3.10/subprocess.py", line 420, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/local/lib/python3.10/subprocess.py", line 524, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command 'pkill -f 30' died with <Signals.SIGTERM: 15>.

(ignore docker and firefox processes here, they're just for completeness here and to demonstrate that you're killing too much processes, ff tabs crash after this)

The interesting output part from pgrep here:

2487379 sleep 30
2487381 /bin/sh -c pgrep -f 30 -la

When you call pkill, it is a process. It will be there instead of pgrep, something like

2487379 sleep 30
2487xxx /bin/sh -c pkill -f 30

... and you pkill all matching processes. So you can kill the running instance successfully, but then kill yourself.

To demonstrate that, the following succeeds:

import subprocess
import time

def run_instance():
    command = "sleep 30 >out 2>err &"
    subprocess.check_output(command, shell=True, start_new_session=True)

def kill_instance():
    print(subprocess.check_output("pgrep -f 'sleep 30' -la", shell=True, universal_newlines=True))
    subprocess.check_output("pgrep -f 'sleep 30' | grep -v $$ | xargs kill -SIGTERM", shell=True, universal_newlines=True)

run_instance()
time.sleep(1)
kill_instance()

Output:

2508440 sleep 30
2508441 /bin/sh -c pgrep -f 'sleep 30' -la

Sorry for that ugly pipe, maybe there is a simpler way, but Linux pgrep cannot exclude self (as opposed to FreeBSD, discussed here). This pipe lists processes, excludes pgrep itself ($$ refers to current parent shell, which is that runs pgrep here) and passes them to kill.

What if I need the managed process (`sleep` in the example) to persist even after the main script dies for any reason (e.g. killed from the terminal)? I was trying to solve that with the `start_new_session` argument to `subprocess.check_output`... — AlwaysLearning, Jul 24 '22 at 16:11
`Popen` is not terminated when your application quits (and even when you quit `bash` seesion, AFAIR) unless joined/killed/terminated explicitly. You can try this code without `kill_instance` call to see this behavior. But then don't use stdout/stdin files, they'll be never closed. — STerliakov, Jul 24 '22 at 16:25
See [this answer](https://stackoverflow.com/a/2251026/14401160) — STerliakov, Jul 24 '22 at 16:38
Augh, sorry, I missed that linux was not explicitly named there. FreeBSD solution from there works on linux for me. — STerliakov, Jul 25 '22 at 11:25

tripleee · Answer 2 · 2022-07-24T16:14:35.707

You are vastly overcomplicating things by basically running shell scripts on top of Python.

Here's (something like) the proper solution.

def run_instance():
    command = ["sleep", "30"]
    return subprocess.Popen(command,
        stdin=subprocess.DEVNULL,
        stdout=subprocess.DEVNULL,
        stderr=subprocess.DEVNULL)

def kill_instance(proc):
    proc.kill()

proc = run_instance()
time.sleep(1)
kill_instance(proc)

A more complete example would proc.wait() if the kill failed for whatever reason.

Unable to pkill a subprocess

2 Answers2

Common way

Your code

Linked