373

I want to use subprocess.check_output() with ps -A | grep 'process_name'. I tried various solutions but so far nothing worked. Can someone guide me how to do it?

martineau
  • 119,623
  • 25
  • 170
  • 301
zuberuber
  • 3,851
  • 2
  • 16
  • 20

8 Answers8

608

To use a pipe with the subprocess module, you have to pass shell=True.

However, this isn't really advisable for various reasons, not least of which is security. Instead, create the ps and grep processes separately, and pipe the output from one into the other, like so:

ps = subprocess.Popen(('ps', '-A'), stdout=subprocess.PIPE)
output = subprocess.check_output(('grep', 'process_name'), stdin=ps.stdout)
ps.wait()

In your particular case, however, the simple solution is to call subprocess.check_output(('ps', '-A')) and then str.find on the output.

Taymon
  • 24,950
  • 9
  • 62
  • 84
  • 136
    +1 for separating the output/input to avoid using `shell=True` – Nicolas Nov 11 '12 at 16:44
  • 10
    Don't forget, error ```subprocess.CalledProcessError: Command '('grep', 'process_name')' returned non-zero exit status 1``` just means that nothing was found by grep, so it's normal behaviour. – Serge Jan 27 '15 at 12:17
  • 6
    Why do we need the `ps.wait()` for when we already have the output. `ps.wait.__doc__` waits for the child to terminate but the content of the child seems already placed into the `output` variable – Papouche Guinslyzinho Aug 25 '15 at 01:07
  • It would be nice to use a Python function instead of grep, but I just found that str.find is (now) considered deprecated: https://docs.python.org/2/library/string.html#deprecated-string-functions – MakisH Oct 16 '15 at 19:54
  • 3
    @MakisH You're looking at `string.find`, which has been deprecated in favor of `str.find` (i.e., the method `find` on `str` objects). – Taymon Oct 16 '15 at 21:20
  • 8
    note: if `grep` dies prematurely; `ps` may hang indefinitely if it produces enough output to fill its OS pipe buffer (because you haven't called `ps.stdout.close()` in the parent). [Swap the starting order, to avoid it](http://stackoverflow.com/a/9164238/4279) – jfs Mar 22 '16 at 17:23
  • 1
    What is the reason for using `ps.wait()` instead of `output.wait()`? – aturegano Apr 18 '17 at 10:31
  • 1
    @aturegano: Because `output` is a bytestring and does not have a `wait` method. – Taymon Apr 18 '17 at 19:17
  • 1
    Some actual explanation for why not to use `shell=True`: https://stackoverflow.com/questions/3172470/actual-meaning-of-shell-true-in-subprocess – User Feb 22 '18 at 07:29
  • If there is multiple pipe operations should I do `ps.wait()` for each command in order? @Taymon – alper Aug 31 '18 at 13:56
  • In my case, I was using `yes | accept_licenses` and since `yes` doesn't terminate on it's own, I had to do `time.wait(10); ps.terminate()` instead of `ps.wait()` – Stan Kurdziel Sep 27 '18 at 18:33
  • Is there a reason you're using tuples instead of lists in the Popen? Why didn't u use `subprocess.Popen(['ps', '-A'], stdout=subprocess.PIPE)` instead of `subprocess.Popen(('ps', '-A'), stdout=subprocess.PIPE)` ? This is the first time I've encountered such usage and I wonder if there's a reason. – eaydin Nov 14 '18 at 18:55
  • @eaydin: Both `list`s and `tuple`s are sequences, so either will work. The latter aren't mutable and are probably created faster (although I've never tested that assumption). In this particular case the difference likely isn't significant. – martineau Jan 07 '19 at 21:48
  • as @jfs said, you missed to close the `ps.stdout`, check the man https://docs.python.org/3.5/library/subprocess.html#replacing-shell-pipeline – andras.tim Sep 29 '20 at 13:33
84

Or you can always use the communicate method on the subprocess objects.

cmd = "ps -A|grep 'process_name'"
ps = subprocess.Popen(cmd,shell=True,stdout=subprocess.PIPE,stderr=subprocess.STDOUT)
output = ps.communicate()[0]
print(output)

The communicate method returns a tuple of the standard output and the standard error.

Guillaume Jacquenot
  • 11,217
  • 6
  • 43
  • 49
jkalivas
  • 1,117
  • 1
  • 8
  • 11
  • 10
    I think using `communicate` is better than `wait`. [There is](https://docs.python.org/2/library/subprocess.html#subprocess.Popen.wait) such warning: "This will deadlock when using stdout=PIPE and/or stderr=PIPE and the child process generates enough output to a pipe such that it blocks waiting for the OS pipe buffer to accept more data. Use communicate() to avoid that." – Paolo Mar 12 '16 at 16:54
  • 6
    To clarify Paolo's comment above, the warning is for wait, not for communicate - i.e. it's the reason he says communicate is better. – EnemyBagJones Dec 02 '16 at 21:16
  • The output of ps.communicate()[0] in python3 returns a bytes object. – Miguel Ortiz Aug 18 '20 at 14:48
  • You are reinventing `subprocess.check_output`, not too poorly but unattractively. As the documentation suggests, you should avoid the low-level `Popen` when the library already provides higher-level functions which take care of all this plumbing in a single line of code, often with better behavior for boundary conditions. – tripleee Jul 11 '21 at 08:11
  • And why are you redirecting standard error to `STDOUT`? – tripleee Jul 11 '21 at 08:12
  • Redirecting standard error to stdout makes sure you catch both stderr and stdout. – JvO Feb 17 '23 at 11:35
42

Using input from subprocess.run you can pass the output of one command into a second one.

import subprocess
    
ps = subprocess.run(['ps', '-A'], check=True, capture_output=True)
processNames = subprocess.run(['grep', 'process_name'],
                              input=ps.stdout, capture_output=True)
print(processNames.stdout.decode('utf-8').strip())
Alan W. Smith
  • 24,647
  • 4
  • 70
  • 96
anaken78
  • 668
  • 5
  • 8
  • 6
    NOTE: `capture_output` will only work for Python 3.7.9 and above. – MightyInSpirit Jan 25 '21 at 16:31
  • 2
    what does `check` do and what's the purpose of `capture_output`? – CervEd May 05 '21 at 21:25
  • 7
    @CervEd Both of these are clearly documented. `capture_output` is a shorthand for the option combination `stdout=supprocess.PIPE, stderr=subprocess.PIPE` and `check=True` raises an error if the subprocess did not return a success (zero) status. – tripleee Jul 11 '21 at 08:17
  • 2
    @tripleee they are documented, somewhere in the unwieldy Python documentation, but there's no detail in the answer as to why they are included. `check=True` is for example not strictly necessary but `capture_output=True` is for the answer to work. The reason for using these options should be included as a part of the answer – CervEd Jul 11 '21 at 09:38
  • I don't necessarily disagree as such, but the answer has a link directly to the pertinent documentation. Perhaps you could have clicked through? – tripleee Jul 11 '21 at 09:41
  • 1
    the python documentation is great man, https://docs.python.org/3/library/subprocess.html ctrl-f "capture_output" – Jules G.M. Jul 29 '21 at 20:35
  • 2
    A downside to this approach is that `capture_output` will read all of the process's stdout into memory. For small programs like ps, this may be fine, but for larger analysis pipelines this should be avoided. – Eldritch Cheese Aug 25 '21 at 11:19
  • @EldritchCheese, would using `stdout=supprocess.PIPE, stderr=subprocess.PIPE` instead of `capture_output=True` avoid this issue? If not, how would you recommend avoiding this issue? Thanks! – perry_the_python Mar 14 '23 at 17:28
  • @perry_the_python The `capture_output=True` flag already sets `stdout=subprocess.PIPE, stderr=subprocess.PIPE` for the internal `Popen` object, so it wouldn't avoid the issue. Instead, you would need to call `subprocess.Popen` directly, and read/process the output incrementally. To the best of my knowledge, there isn't a straight-forward way to do this with `subprocess.run`, and so it's better to use `subprocess.Popen` directly in this case. – Eldritch Cheese Mar 26 '23 at 15:17
31

See the documentation on setting up a pipeline using subprocess: http://docs.python.org/2/library/subprocess.html#replacing-shell-pipeline

I haven't tested the following code example but it should be roughly what you want:

query = "process_name"
ps_process = Popen(["ps", "-A"], stdout=PIPE)
grep_process = Popen(["grep", query], stdin=ps_process.stdout, stdout=PIPE)
ps_process.stdout.close()  # Allow ps_process to receive a SIGPIPE if grep_process exits.
output = grep_process.communicate()[0]
AlcubierreDrive
  • 3,654
  • 2
  • 29
  • 45
4

You can try the pipe functionality in sh.py:

import sh
print sh.grep(sh.ps("-ax"), "process_name")
cyraxjoe
  • 5,661
  • 3
  • 28
  • 42
amoffat
  • 696
  • 4
  • 12
3

Also, try to use 'pgrep' command instead of 'ps -A | grep 'process_name'

Shooe
  • 75
  • 2
2
command = "ps -A | grep 'process_name'"
output = subprocess.check_output(["bash", "-c", command])
Brent
  • 4,153
  • 4
  • 30
  • 63
  • 3
    Why not use `shell=True` and let that prepend `['sh', '-c']`? Nothing in this code requires bash. (That said, it's significantly better practice to avoid using a shell at all; this use case is a reasonable one, but as soon as arguments start to get parameterized -- like taking the process_name as a parameter -- security concerns come in). – Charles Duffy Sep 03 '20 at 16:39
  • It's useful in that you don't have to split the string, which gets complicated when you have quoted white space. – Brent Sep 03 '20 at 22:09
  • 1
    Huh? `subprocess.check_output(command, shell=True)` doesn't require you to split the string. `Popen` converts any string into a list containing only that string -- thus, `[command]` -- so with `shell=True` you get `['sh', '-c']` prepended to that list, so you end up with `['sh', '-c', command]`, exactly what your code does here except for the `sh`/`bash` difference. – Charles Duffy Sep 03 '20 at 22:10
  • 1
    ...for that matter, if you _did_ try to split the string into a list _as well as_ using `shell=True`, only the first element of that list would be treated as code; you'd get something like `['sh', '-c', 'ps', '-A', '|', 'grep', 'process_name']`. That's not a useful thing to do: when invoked that way, the shell runs `ps` with `$0` being `-A`, `$1` being `|`, etc... but since the command `ps` doesn't look at `$0`, `$1`, etc., all that extra content is simply ignored. – Charles Duffy Sep 03 '20 at 22:13
  • I wish I could provide a counterexample. This answer exists because I found one, but I don't remember it now. Doing it like this worked perfectly. – Brent Sep 04 '20 at 00:11
  • 2
    If you read [`Lib/subprocess.py`](https://github.com/python/cpython/blob/cdbff3527c5b10e79761ad25316b46cd079304bc/Lib/subprocess.py#L1680-L1686), you'll see that there _literally is no difference_ between `subprocess.check_output(["sh", "-c", command])` and `subprocess.check_output(command, shell=True)`. The code is clear and simple -- this is not a place where there can be a devil hiding in the details somewhere. – Charles Duffy Sep 04 '20 at 01:30
  • Charles is hitting hard with knowledge. – Baskaya Nov 13 '20 at 05:51
0

I think that launching a shell just to enjoy the pipelining is not as elegant as it could be.

The following code uses native subprocess pipeline support and it works indeed.

You could easily modify it to add more than two processes to the pipeline.

#!/usr/bin/env python3

import subprocess


def ps_grep(pattern):
    # First command-line
    ps_command = ["ps", "-A"]

    # Second command-line
    grep_command = ["grep", pattern]

    # Launch first process
    ps_process = subprocess.Popen(ps_command, stdout=subprocess.PIPE)

    # Launch second process and connect it to the first one
    grep_process = subprocess.Popen(
        grep_command, stdin=ps_process.stdout, stdout=subprocess.PIPE
    )

    # Let stream flow between them
    output, _ = grep_process.communicate()

    return output.decode()


if __name__ == "__main__":
    print(ps_grep("python"))
Raúl Salinas-Monteagudo
  • 3,412
  • 1
  • 24
  • 22