Store output of subprocess.Popen call in a string

Question

I'm trying to make a system call in Python and store the output to a string that I can manipulate in the Python program.

#!/usr/bin/python
import subprocess
p2 = subprocess.Popen("ntpq -p")

I've tried a few things including some of the suggestions here:

Retrieving the output of subprocess.call()

but without any luck.

Answer 1

In Python 2.7 or Python 3

Instead of making a Popen object directly, you can use the subprocess.check_output() function to store output of a command in a string:

from subprocess import check_output
out = check_output(["ntpq", "-p"])

In Python 2.4-2.6

Use the communicate method.

import subprocess
p = subprocess.Popen(["ntpq", "-p"], stdout=subprocess.PIPE)
out, err = p.communicate()

out is what you want.

Important note about the other answers

Note how I passed in the command. The "ntpq -p" example brings up another matter. Since Popen does not invoke the shell, you would use a list of the command and options—["ntpq", "-p"].

Answer 2

This worked for me for redirecting stdout (stderr can be handled similarly):

from subprocess import Popen, PIPE
pipe = Popen(path, stdout=PIPE)
text = pipe.communicate()[0]

If it doesn't work for you, please specify exactly the problem you're having.

Answer 3

In Python 3.7+ you can use the new capture_output= keyword argument for subprocess.run:

import subprocess

p = subprocess.run(["echo", "hello world!"], capture_output=True, text=True)
assert p.stdout == 'hello world!\n'

Answer 4

Python 2: http://docs.python.org/2/library/subprocess.html#subprocess.Popen

from subprocess import PIPE, Popen

command = "ntpq -p"
process = Popen(command, stdout=PIPE, stderr=None, shell=True)
output = process.communicate()[0]
print output

In the Popen constructor, if shell is True, you should pass the command as a string rather than as a sequence. Otherwise, just split the command into a list:

command = ["ntpq", "-p"]
process = Popen(command, stdout=PIPE, stderr=None)

If you need to read also the standard error, into the Popen initialization, you should set stderr to PIPE or STDOUT:

command = "ntpq -p"
process = subprocess.Popen(command, stdout=PIPE, stderr=PIPE, shell=True)
output, error = process.communicate()

NOTE: Starting from Python 2.7, you could/should take advantage of subprocess.check_output (https://docs.python.org/2/library/subprocess.html#subprocess.check_output).

---

Python 3: https://docs.python.org/3/library/subprocess.html#subprocess.Popen

from subprocess import PIPE, Popen

command = "ntpq -p"
with Popen(command, stdout=PIPE, stderr=None, shell=True) as process:
    output = process.communicate()[0].decode("utf-8")
    print(output)

NOTE: If you're targeting only versions of Python higher or equal than 3.5, then you could/should take advantage of subprocess.run (https://docs.python.org/3/library/subprocess.html#subprocess.run).

Answer 5

Assuming that pwd is just an example, this is how you can do it:

import subprocess

p = subprocess.Popen("pwd", stdout=subprocess.PIPE)
result = p.communicate()[0]
print result

See the subprocess documentation for another example and more information.

Answer 6

for Python 2.7+ the idiomatic answer is to use subprocess.check_output()

You should also note the handling of arguments when invoking a subprocess, as it can be a little confusing....

If args is just single command with no args of its own (or you have shell=True set), it can be a string. Otherwise it must be a list.

for example... to invoke the ls command, this is fine:

from subprocess import check_call
check_call('ls')

so is this:

from subprocess import check_call
check_call(['ls',])

however, if you want to pass some args to the shell command, you can't do this:

from subprocess import check_call
check_call('ls -al')

instead, you must pass it as a list:

from subprocess import check_call
check_call(['ls', '-al'])

the shlex.split() function can sometimes be useful to split a string into shell-like syntax before creating a subprocesses... like this:

from subprocess import check_call
import shlex
check_call(shlex.split('ls -al'))

Answer 7

This works perfectly for me:

import subprocess
try:
    #prints results and merges stdout and std
    result = subprocess.check_output("echo %USERNAME%", stderr=subprocess.STDOUT, shell=True)
    print result
    #causes error and merges stdout and stderr
    result = subprocess.check_output("copy testfds", stderr=subprocess.STDOUT, shell=True)
except subprocess.CalledProcessError, ex: # error code <> 0 
    print "--------error------"
    print ex.cmd
    print ex.message
    print ex.returncode
    print ex.output # contains stdout and stderr together 

Answer 8

This was perfect for me. You will get the return code, stdout and stderr in a tuple.

from subprocess import Popen, PIPE

def console(cmd):
    p = Popen(cmd, shell=True, stdout=PIPE)
    out, err = p.communicate()
    return (p.returncode, out, err)

For Example:

result = console('ls -l')
print 'returncode: %s' % result[0]
print 'output: %s' % result[1]
print 'error: %s' % result[2]

Answer 9

The accepted answer is still good, just a few remarks on newer features. Since python 3.6, you can handle encoding directly in check_output, see documentation. This returns a string object now:

import subprocess 
out = subprocess.check_output(["ls", "-l"], encoding="utf-8")

In python 3.7, a parameter capture_output was added to subprocess.run(), which does some of the Popen/PIPE handling for us, see the python docs :

import subprocess 
p2 = subprocess.run(["ls", "-l"], capture_output=True, encoding="utf-8")
p2.stdout

Answer 10

I wrote a little function based on the other answers here:

def pexec(*args):
    return subprocess.Popen(args, stdout=subprocess.PIPE).communicate()[0].rstrip()

Usage:

changeset = pexec('hg','id','--id')
branch = pexec('hg','id','--branch')
revnum = pexec('hg','id','--num')
print('%s : %s (%s)' % (revnum, changeset, branch))

Answer 11

 import os   
 list = os.popen('pwd').read()

In this case you will only have one element in the list.

Answer 12

import subprocess
output = str(subprocess.Popen("ntpq -p",shell = True,stdout = subprocess.PIPE, 
stderr = subprocess.STDOUT).communicate()[0])

This is one line solution

Answer 13

The following captures stdout and stderr of the process in a single variable. It is Python 2 and 3 compatible:

from subprocess import check_output, CalledProcessError, STDOUT

command = ["ls", "-l"]
try:
    output = check_output(command, stderr=STDOUT).decode()
    success = True 
except CalledProcessError as e:
    output = e.output.decode()
    success = False

If your command is a string rather than an array, prefix this with:

import shlex
command = shlex.split(command)

Answer 14

Use check_output method of subprocess module

import subprocess

address = '192.168.x.x'
res = subprocess.check_output(['ping', address, '-c', '3'])

Finally parse the string

for line in res.splitlines():

Hope it helps, happy coding

Answer 15

For python 3.5 I put up function based on previous answer. Log may be removed, thought it's nice to have

import shlex
from subprocess import check_output, CalledProcessError, STDOUT


def cmdline(command):
    log("cmdline:{}".format(command))
    cmdArr = shlex.split(command)
    try:
        output = check_output(cmdArr,  stderr=STDOUT).decode()
        log("Success:{}".format(output))
    except (CalledProcessError) as e:
        output = e.output.decode()
        log("Fail:{}".format(output))
    except (Exception) as e:
        output = str(e);
        log("Fail:{}".format(e))
    return str(output)


def log(msg):
    msg = str(msg)
    d_date = datetime.datetime.now()
    now = str(d_date.strftime("%Y-%m-%d %H:%M:%S"))
    print(now + " " + msg)
    if ("LOG_FILE" in globals()):
        with open(LOG_FILE, "a") as myfile:
            myfile.write(now + " " + msg + "\n")