Retrieving the output of subprocess.call()

Question

How can I get the output of a process run using subprocess.call()?

Passing a StringIO.StringIO object to stdout gives this error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/subprocess.py", line 444, in call
    return Popen(*popenargs, **kwargs).wait()
  File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/subprocess.py", line 588, in __init__
    errread, errwrite) = self._get_handles(stdin, stdout, stderr)
  File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/subprocess.py", line 945, in _get_handles
    c2pwrite = stdout.fileno()
AttributeError: StringIO instance has no attribute 'fileno'
>>>

Mike's answer is correct. Note that `StringIO` works like a file *in most cases* but not all. It doesn't work in your case because the `multiprocessing` module assumes actual files in some cases. This may have been fixed: see http://bugs.python.org/issue5313 for a related bug. — Michael Greene, Jan 03 '10 at 22:16
Actually, `communicate()` uses `select.select()`, which only accepts file descriptors, so it isn't really a bug. I was quite confused by this when I first encountered it and exploring the depths of subprocess.py taught me a lot!. — Mike, Jan 03 '10 at 22:25
I think [`subprocess.run`](https://docs.python.org/3/library/subprocess.html#subprocess.run) makes this simpler, as of Python 3.5. I'll add an answer when I get a chance. — Mark Amery, Apr 11 '16 at 09:36
Notice tha the accepted answer is way obsolete. The simple answer for Python 2.7 would be `subprocess.check_output()`; in Python 3.5+ you will also want to look at `subprocess.run()`. There should be no need or want to use raw `subprocess.Popen()` if you can avoid it, though some more complex use cases require it (and then you have to do the required plumbing around it yourself). The [Stack Overflow `subprocess` tag info page](/tags/subprocess/info) has some good resources for the less trivial cases. — tripleee, Feb 05 '21 at 09:50

score 337 · Answer 1 · edited Jul 08 '22 at 13:20

337

If you have Python version >= 2.7, you can use subprocess.check_output which basically does exactly what you want (it returns standard output as string).

Simple example (linux version, see note):

import subprocess

print subprocess.check_output(["ping", "-c", "1", "8.8.8.8"])

Note that the ping command is using linux notation (-c for count). If you try this on Windows remember to change it to -n for same result.

As commented below you can find a more detailed explanation in this other answer.

edited Jul 08 '22 at 13:20

Neuron

5,141
5
38
59

answered Jan 02 '12 at 11:45

sargue

5,695
3
28
43

42

[Found another answer with working code](http://stackoverflow.com/a/8235171/881224). Please upvote if you used it. – yurisich Jul 19 '12 at 13:17
9

But be aware that check_output will throw an exception if the process returns a non zero exit code. – tobltobs Aug 04 '15 at 15:35
Note that `check_output` suffers from the PIPE filling up problem as `run` since it's just calling `run`. So if you have a process generating a bit more output it will hang indefinitely. The `Popen` solution from [@Mike's](https://stackoverflow.com/a/1996540/220986) and [@Jabba's](https://stackoverflow.com/a/21000308/220986) answers works a lot better – ChrisWue May 26 '17 at 03:26
But the other answer does not demonstrate how to read stdout from check_call. – Ray Salemi Sep 28 '17 at 20:48
You should really put it in your answer though. – Big McLargeHuge Mar 07 '18 at 15:34
What is stored in the output? If running, for example a shell script then is it the value in echo / cat command or something else? – arqam Sep 25 '18 at 09:15
it is what would normally be printed in binary. you can do `subprocess.check_output(['ls", "/"]).decode()` for example – ozgeneral Feb 20 '19 at 15:27
check_output also has the barrier that it does a "check." In my current use case, I actually need to ignore non-zero return values. – macetw Jun 16 '20 at 16:08

score 233 · Accepted Answer · edited May 06 '14 at 12:33

233

Output from subprocess.call() should only be redirected to files.

You should use subprocess.Popen() instead. Then you can pass subprocess.PIPE for the stderr, stdout, and/or stdin parameters and read from the pipes by using the communicate() method:

from subprocess import Popen, PIPE

p = Popen(['program', 'arg1'], stdin=PIPE, stdout=PIPE, stderr=PIPE)
output, err = p.communicate(b"input data that is passed to subprocess' stdin")
rc = p.returncode

The reasoning is that the file-like object used by subprocess.call() must have a real file descriptor, and thus implement the fileno() method. Just using any file-like object won't do the trick.

See here for more info.

edited May 06 '14 at 12:33

jfs

399,953
195
994
1,670

answered Jan 03 '10 at 22:13

Mike

4,976
2
18
11

60

this page http://docs.python.org/library/subprocess.html#module-subprocess discourages using `subprocess.PIPE`, any idea how to overcome this? – Vladimir Keleshev Dec 13 '11 at 20:55
6

also, the question especifies using subprocess.call and Mike's answer is using Popen in fact, as subprocess.call only return the returncode, but no means of accessing any of the streams. That's if using 2.6, if using 2.7 @Sergi answer could be used – Willyfrog Nov 12 '12 at 12:17
This seems to be an incorrect suggestion... – László Papp Nov 15 '13 at 14:22
20

@Halst: the docs warn about `PIPE` for the `call()` call (don't use `PIPE` in this case). It is fine to use `PIPE` with `subprocess.Popen` e.g., `output, _ = Popen(..., stdout=PIPE).communicate()` as this answer suggests. – jfs Jan 10 '14 at 22:42
// , I can confirm that this works. @J.F.Sebastian, what difference between subprocess.call() and subprocess.Popen() makes PIPE less secure for the former? – Nathan Basanese Sep 02 '15 at 23:20
6

@NathanBasanese: in short: doesn't use `PIPE` unless you consume the pipe. `call()` is `Popen().wait()` and therefore it does not consume the pipes (as soon as the corresponding OS pipe buffer fills, the child process will hang forever). `Popen().communicate()` writes/reads data from pipes if `PIPE` is used thus allowing the child process to continue. – jfs Sep 02 '15 at 23:32
2

// , Ah, OK. That makes sense. Weird that it even allows PIPE as an argument, then. Anyway, I was a good StackOverFlow citizen, though, and made a question for it: http://stackoverflow.com/questions/32364849/what-difference-between-subprocess-call-and-subprocess-popen-makes-pipe would you be willing to throw that in as the answer? – Nathan Basanese Sep 02 '15 at 23:41
1

What is the b for in `p.communicate(b"in`? – clankill3r Mar 08 '17 at 19:24
1

@clankill3r it makes a Bytes literal instead of a string literal in Python 3. https://stackoverflow.com/questions/6269765/what-does-the-b-character-do-in-front-of-a-string-literal – Clay Jun 16 '17 at 20:50
No, it's not working with Python 2.7.15. Empty `output`. – Arkady Aug 07 '18 at 06:02

chtenb · Answer 3 · 2016-01-21T17:13:40.053

103

For python 3.5+ it is recommended that you use the run function from the subprocess module. This returns a CompletedProcess object, from which you can easily obtain the output as well as return code.

from subprocess import PIPE, run

command = ['echo', 'hello']
result = run(command, stdout=PIPE, stderr=PIPE, universal_newlines=True)
print(result.returncode, result.stdout, result.stderr)

edited Jan 21 '16 at 17:13

answered Jan 19 '16 at 09:44

chtenb

14,924
14
78
116

This is a brilliant answer. But don't overlook the 'stdout=PIPE' part, as I did the first time around, otherwise the output will be very shortlived! – excyberlabber Jul 28 '17 at 09:31
3

Instead of ``universal_newlines=True`` you should use ``text=True`` as the former is kept for backwards compatibility. – tobiasraabe May 08 '19 at 11:34
4

"It's recommended to use the `run` function" - that doesn't work for me when I use those parameters, it seems to be a problem with Windows. I solved it by adding `shell=True` to the `run` command! And regardless of what OS you're using, if you have Python 3.7+, replace `stdout=PIPE, stderr=PIPE` with `capture_output=True` to add a buffer that won't get deadlocked. – Post169 Oct 02 '19 at 21:51
3

@Post169 Great answer for the Python 3.7+ crowd! So I ran `rc = subprocess.run(ogrList, capture_output=True)` then `print("\n", rc.stdout)` and it worked great! Thanks! – grego Sep 19 '20 at 01:10

Jabba · Answer 4 · 2014-01-10T09:57:36.760

53

I have the following solution. It captures the exit code, the stdout, and the stderr too of the executed external command:

import shlex
from subprocess import Popen, PIPE

def get_exitcode_stdout_stderr(cmd):
    """
    Execute the external command and get its exitcode, stdout and stderr.
    """
    args = shlex.split(cmd)

    proc = Popen(args, stdout=PIPE, stderr=PIPE)
    out, err = proc.communicate()
    exitcode = proc.returncode
    #
    return exitcode, out, err

cmd = "..."  # arbitrary external command, e.g. "python mytest.py"
exitcode, out, err = get_exitcode_stdout_stderr(cmd)

I also have a blog post on it here.

Edit: the solution was updated to a newer one that doesn't need to write to temp. files.

edited Jan 10 '14 at 09:57

answered Jan 08 '14 at 15:50

Jabba

19,598
6
52
45

2

@Jabba for some reason it wouldn't work unless I added `shell=True` to the arguments of Popen(), could you explain why? – JaeGeeTee Feb 18 '14 at 07:47
@JaeGeeTee: What is the command you wanted to call? My guess is that you wanted to call a command that contains pipes (e.g. "`cat file.txt | wc -l`"). – Jabba Feb 20 '14 at 16:57
@Jabba I was running `ping 127.0.0.1` – JaeGeeTee Feb 20 '14 at 23:15
@JaeGeeTee - try using /bin/ping explicitly? – cdyson37 Apr 24 '14 at 10:14
Doesn't really answer the question of how to use call or check_call for a very small output. – Ray Salemi Sep 28 '17 at 20:50
dont we first have to proc.wait() before reading out and err? – infoclogged Oct 06 '17 at 16:49

score 31 · Answer 5 · edited Aug 27 '20 at 20:45

I recently just figured out how to do this, and here's some example code from a current project of mine:

#Getting the random picture.
#First find all pictures:
import shlex, subprocess
cmd = 'find ../Pictures/ -regex ".*\(JPG\|NEF\|jpg\)" '
#cmd = raw_input("shell:")
args = shlex.split(cmd)
output,error = subprocess.Popen(args,stdout = subprocess.PIPE, stderr= subprocess.PIPE).communicate()
#Another way to get output
#output = subprocess.Popen(args,stdout = subprocess.PIPE).stdout
ber = raw_input("search complete, display results?")
print output
#... and on to the selection process ...

You now have the output of the command stored in the variable "output". "stdout = subprocess.PIPE" tells the class to create a file object named 'stdout' from within Popen. The communicate() method, from what I can tell, just acts as a convenient way to return a tuple of the output and the errors from the process you've run. Also, the process is run when instantiating Popen.

Doesn't really answer the question of how to use call (judiciously.) — Ray Salemi, Sep 28 '17 at 20:51

score 29 · Answer 6 · edited Nov 24 '21 at 18:22

The key is to use the function subprocess.check_output

For example, the following function captures stdout and stderr of the process and returns that as well as whether or not the call succeeded. It is Python 2 and 3 compatible:

from subprocess import check_output, CalledProcessError, STDOUT

def system_call(command):
    """ 
    params:
        command: list of strings, ex. `["ls", "-l"]`
    returns: output, success
    """
    try:
        output = check_output(command, stderr=STDOUT).decode()
        success = True 
    except CalledProcessError as e:
        output = e.output.decode()
        success = False
    return output, success

output, success = system_call(["ls", "-l"])

If you want to pass commands as strings rather than arrays, use this version:

from subprocess import check_output, CalledProcessError, STDOUT
import shlex

def system_call(command):
    """ 
    params:
        command: string, ex. `"ls -l"`
    returns: output, success
    """
    command = shlex.split(command)
    try:
        output = check_output(command, stderr=STDOUT).decode()
        success = True 
    except CalledProcessError as e:
        output = e.output.decode()
        success = False
    return output, success

output, success = system_call("ls -l")

Np, was easy to search. BTW, it also has an `input=` option to specify input to pipe into the command instead of that super-overly-complicated way of doing it with `Popen`. — sudo, Nov 17 '17 at 22:16
Absolutely great way of doing that, exactly what I was looking for. And to clarify on Jason's comment: with non-zero exit code you'll get to exception handler part, so your `success` will be `False` — retif, Aug 31 '20 at 14:46

score 16 · Answer 7 · answered Sep 05 '15 at 13:14

16

In Ipython shell:

In [8]: import subprocess
In [9]: s=subprocess.check_output(["echo", "Hello World!"])
In [10]: s
Out[10]: 'Hello World!\n'

Based on sargue's answer. Credit to sargue.

answered Sep 05 '15 at 13:14

jhegedus

20,244
16
99
167

1

`s=subprocess.check_output(["echo", "Hello World!"]); print(s)` prints as `b'Hello World!\n'` how could I get rid of `b''`index ? @jhegedus – alper Aug 31 '18 at 12:50
1

try b'Hello World!\n'.decode("utf-8") – Soumyajit Aug 18 '19 at 08:32

Retrieving the output of subprocess.call()

7 Answers7

Linked

Related