53

I have a case to want to execute the following shell command in Python and get the output,

echo This_is_a_testing | grep -c test

I could use this python code to execute the above shell command in python,

>>> import subprocess
>>> subprocess.check_output("echo This_is_a_testing | grep -c test", shell=True)
'1\n'

However, as I do not want to use the "shell=True" option, I tried the following python code,

>>> import subprocess
>>> p1 = subprocess.Popen(["echo", "This_is_a_testing"], stdout=subprocess.PIPE)
>>> p2 = subprocess.Popen(["grep", "-c", "test"], stdin=p1.stdout)
>>> p1.stdout.close()
>>> p2.communicate()
(None, None)

I wonder why the output is "None" as I have referred to the descriptions in the webpage : http://docs.python.org/library/subprocess.html#subprocess.PIPE

Had I missed some points in my code ? Any suggestion / idea ? Thanks in advance.

qneill
  • 1,643
  • 14
  • 18
user1129812
  • 2,859
  • 8
  • 32
  • 45
  • related: [How do I use subprocess.Popen to connect multiple processes by pipes?](http://stackoverflow.com/q/295459/4279) – jfs Sep 14 '15 at 16:27

5 Answers5

42
>>> import subprocess

>>> mycmd=subprocess.getoutput('df -h | grep home | gawk \'{ print $1 }\' | cut -d\'/\' -f3')

>>> mycmd 

'sda6'

>>>
Wtower
  • 18,848
  • 11
  • 103
  • 80
Дмитрий Голуб
  • 437
  • 4
  • 2
  • 2
    it uses the shell internally. OP explicitly said: *"I do not want to use the "shell=True" option"* – jfs Sep 14 '15 at 16:27
  • 18
    Thanks. That's exactly what I'm looking for when searching for "***how to execute shell commands with pipes***" in Python. – xpt Jul 14 '16 at 21:46
  • 4
    I had to use commands.getoutput instead of subprocess.getoutput for this to work – Anake Nov 29 '16 at 12:12
  • 1
    Isn't is `subprocess.check_output()`? – WesternGun Oct 20 '17 at 08:43
  • 1
    Doesn't exist in Python 2.7, does in Python 3.6.6 – DavidJ Dec 06 '18 at 17:29
  • Holy cr@p this was helpful! I had been wracking my brain with sed, perl, awk, node, python, etc... trying to do a string substitution where the replacement contains the output of piped shell commands for each match. [My example gist](https://gist.github.com/wickkidd/7d9c87d7459b58f1e0a6df71906cb234) – bkidd Mar 19 '19 at 15:25
  • You can also do `subprocess.getstatusoutput` which returns a tuple, where the first element is the exit code. – Rok Povsic Jan 03 '23 at 08:54
39

Please look here:

>>> import subprocess
>>> p1 = subprocess.Popen(["echo", "This_is_a_testing"], stdout=subprocess.PIPE)
>>> p2 = subprocess.Popen(["grep", "-c", "test"], stdin=p1.stdout)
>>> 1
p1.stdout.close()
>>> p2.communicate()
(None, None)
>>>

here you get 1 as output after you write p2 = subprocess.Popen(["grep", "-c", "test"], stdin=p1.stdout), Do not ignore this output in the context of your question.

If this is what you want, then pass stdout=subprocess.PIPE as argument to the second Popen:

>>> p1 = subprocess.Popen(["echo", "This_is_a_testing"], stdout=subprocess.PIPE)
>>> p2 = subprocess.Popen(["grep", "test"], stdin=p1.stdout, stdout=subprocess.PIPE)
>>> p2.communicate()
('This_is_a_testing\n', None)
>>>
Nawaz
  • 353,942
  • 115
  • 666
  • 851
avasal
  • 14,350
  • 4
  • 31
  • 47
  • Now I know that for `p2.communicate()` to work, I must add `stdout=subprocess.PIPE` in the code `p2 = ...`. Thanks. – user1129812 Feb 22 '12 at 12:06
  • How I can get error if it happens in p1 result? Here is my question, can you help with it? http://stackoverflow.com/questions/40467105/how-to-catch-ping-error-with-python – prosto.vint Nov 07 '16 at 17:15
33

From the manual:

to get anything other than None in the result tuple, you need to give stdout=PIPE and/or stderr=PIPE

p2 = subprocess.Popen(["grep", "-c", "test"], stdin=p1.stdout, stdout=subprocess.PIPE)
Karoly Horvath
  • 94,607
  • 11
  • 117
  • 176
6

While the accepted answer is correct/working, another option would be to use the Popen.communicate() method to pass something to a process' stdin:

>>> import subprocess
>>> p2 = subprocess.Popen(["grep", "-c", "test"], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
>>> p2.communicate("This_is_a_testing")
('1\n', None)
>>> print p2.returncode
0
>>>>

This resolves the need to execute another command just to redirect it's output, if the output is already known in the python script itself.

However communicate has the side-effect, that it waits for the process to terminate. If asynchronous execution is needed/desired using two processes might be the better option.

dpr
  • 10,591
  • 3
  • 41
  • 71
1

Answer is similar to mentioned earlier, with little formatting. I wanted to get exactly same output as normal shell command with pipe on python 3.

import subprocess

p1 = subprocess.Popen(["ls", "-l", "."], stdout=subprocess.PIPE)
p2 = subprocess.Popen(["grep", "May"], stdin=p1.stdout, stdout=subprocess.PIPE)


for s in (str(p2.communicate())[2:-10]).split('\\n'):
    print(s)
Naveen Singh
  • 81
  • 4