39

How do I run a command with a pipe | in it?

The subprocess module seems complex...

Is there something like

output,error = `ps cax | grep something`

as in shell script?

Martin Thoma
  • 124,992
  • 159
  • 614
  • 958
eugene
  • 39,839
  • 68
  • 255
  • 489
  • Don't do this with `subprocess`. It's much easier to do this with the shell. Indeed, this is the one thing the shell does best. – S.Lott Jul 21 '11 at 17:19

5 Answers5

71

See Replacing shell pipeline:

import subprocess

proc1 = subprocess.Popen(['ps', 'cax'], stdout=subprocess.PIPE)
proc2 = subprocess.Popen(['grep', 'python'], stdin=proc1.stdout,
                         stdout=subprocess.PIPE, stderr=subprocess.PIPE)

proc1.stdout.close() # Allow proc1 to receive a SIGPIPE if proc2 exits.
out, err = proc2.communicate()
print('out: {0}'.format(out))
print('err: {0}'.format(err))

PS. Using shell=True can be dangerous. See for example the warning in the docs.

There is also the sh module which can make subprocess scripting in Python a lot more pleasant:

import sh
print(sh.grep(sh.ps("cax"), 'something'))
unutbu
  • 842,883
  • 184
  • 1,785
  • 1,677
  • 2
    Great reference to the sh module, I highly recommend it. – jose.angel.jimenez Dec 18 '15 at 12:46
  • 3
    `sh.grep(sh.ps('aux', _piped=True), 'something')` - worked for me – Ben Usman Jun 03 '16 at 05:42
  • @unutbu, Could you explain more detail why `proc1.stdout.close()` should be called? – SangminKim Dec 27 '17 at 18:45
  • 1
    @SangminKim: This appears to have been fixed or worked-around in Python3, but in Python2.7 at least, if you were to spawn a long-running process such as `find / -print` in `proc1` without calling `proc1.stdout.close()` and spawn a short-running command like `head` in `proc2`, then you would see `find / -print` is still running long after the call to `head` has ended. Including `proc1.stdout.close()` allows `find` to end soon after `head` finishes. – unutbu Dec 28 '17 at 02:15
  • 1
    For more on SIGPIPE, see [this comment](https://stackoverflow.com/questions/108183/how-to-prevent-sigpipes-or-handle-them-properly#comment72630256_32597537), [this post](http://www.pixelbeat.org/programming/sigpipe_handling.html) and [this one too](https://stackoverflow.com/q/8369506/190597). – unutbu Dec 28 '17 at 02:15
  • @unutbu, Thank you for your answering. However, I am not clear on that. Thus I open another question on [here](https://stackoverflow.com/q/48001819/3523935), since It is too long to write as a comment. – SangminKim Dec 28 '17 at 05:10
  • ImportError: No module named sh – basickarl Feb 11 '19 at 12:29
24

You've already accepted an answer, but:

Do you really need to use grep? I'd write something like:

import subprocess
ps = subprocess.Popen(('ps', 'cax'), stdout=subprocess.PIPE)
output = ps.communicate()[0]
for line in output.split('\n'):
    if 'something' in line:
        ...

This has the advantages of not involving shell=True and its riskiness, doesn't fork off a separate grep process, and looks an awful lot like the kind of Python you'd write to process data file-like objects.

Kirk Strauser
  • 30,189
  • 5
  • 49
  • 65
12
import subprocess

process = subprocess.Popen("ps cax | grep something",
                             shell=True,
                             stdout=subprocess.PIPE,
                           )
stdout_list = process.communicate()[0].split('\n')
instigator
  • 1,587
  • 5
  • 15
  • 27
  • something in grep something would be a variable, should I construct the "ps cax | grep something" string with variable to run it? or is there other way of doing it? – eugene Jul 21 '11 at 17:23
  • 1
    http://docs.python.org/library/stdtypes.html#string-formatting or http://docs.python.org/library/string.html#string-formatting. Basically, `'grep %s` % variable`. – agf Jul 21 '11 at 17:37
  • 3
    @Eugene: You can construct the string using a variable, but be careful about where the variable is coming from. I.e. make sure it's not from a user who could make "something" into `"something; rm -rf /"`. Building expressions to run with shell=True is a possible security risk. – Thomas K Jul 21 '11 at 17:39
6

Drop that 'ps' subprocess and back away slowly! :)

Use the psutil module instead.

Michael Kent
  • 1,736
  • 12
  • 11
4
import os

os.system('ps -cax|grep something')

If you wanna replace grep argument with some variable:

os.system('ps -cax|grep '+your_var)
Mark Ma
  • 1,342
  • 3
  • 19
  • 35
  • @Kirk Strauser, can't you get the output? I have testd it both in interactive python interpreter and running it as script, it works as expected(my python version is 2.7.1,if it helps) – Mark Ma Jul 22 '11 at 13:01
  • 4
    `os.system` only returns the int exit code of the subprocess. If you're running it at an interactive prompt and seeing the output of ps, it's because ps is writing to stout. Python isn't actually capturing that output. Try it yourself: run `a = os.system('ls')`. You'll still see the output of ls, and `a` will be 0 (assuming ls didn't fail for some reason). – Kirk Strauser Jul 22 '11 at 16:05
  • This is unsafe. If `your_var` is set by someone else, this code lets them run any command they want. For example `your_var = "a; echo you got hacked"` – Boris Verkhovskiy Dec 27 '19 at 04:16