Most pythonic way to get output from a shell command

Question

I'm writing my first python program and I want to run a shell command and get the output. I want to do it in the cleanest possible / most pythonic way. This is what I've got atm. Note: I also want to raise an Exception when there went something worng with execution of the shell command.

def lsCommand():
    process = subprocess.Popen(['ls', '-a'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    out , err = process.communicate()
    returnc = process.returncode
    if returnc != 0:
        raise Exception(err)
    else:
        return out

What I discovered, is that when I remove the line out , err = process.communicate(), and replace it with time.sleep(4) (<= the ls command doesn't take 4 seconds to complete), The returncode is None, which means that it the subprocess isn't fnished yet. How is this possible? Does the out , err = process.communicate() do an implicit wait() methode call?

def lsCommand():
    process = subprocess.Popen(['ls', '-a'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    time.sleep(4)
    returnc = process.returncode
    if returnc != 0:
        print returnc
        raise Exception(err)
    else:
        return out

EDIT: I'm using python 2.7

I want to do a longer command sudo find "/media/usb0" -type f -name "*.JPG" | wc -l, but i'm not sure how to call it with Popen, can anyone help?

Antti Haapala -- Слава Україні · Accepted Answer · 2014-07-25T09:32:43.320

4

Yes, communicate implies wait; it reads the process' stdout and stderr until they are closed.

In the second example you PIPE the output of the ls; the child process ls is blocked writing to stdout and stderr until you read from the pipe - thus it would not completed within 4 seconds, or even 4 years until these descriptors are read. However, as polling the descriptors manually is prone to errors, the Popen class has the communicate method to avoid the pitfalls.

For a very pythonic way to easily get the output of a command, see Popen.check_output available since Python 3.1, and 2.7, though it does not raise an exception with the contents of stderr on failure. Your first example is good if you need the exception to contain the contents of stderr as a message.

To call the full pipeline in Python, you can use the shell=True with the full pipeline as a command.

pipeline = subprocess.Popen(
    'sudo find "/media/usb0" -type f -name "*.JPG" | wc -l',
    shell=True,
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE
)

However, you can also use Popens to make the pipeline:

find = subprocess.Popen(
    ['sudo', 'find', '/media/usb0', '-type', 'f', '-name', '*.JPG'],
    stdout=subprocess.PIPE
)
wc = subprocess.Popen(
    ['wc', '-l'],
    stdin=find.stdout,
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE
)
find.stdout.close()
wc.communicate()

edited Jul 25 '14 at 09:32

answered Jul 25 '14 at 08:53

Antti Haapala -- Слава Україні

129,958
22
279
321

I'm using python 2.7, should have said that. Sorry. I will edit my question – Bosiwow Jul 25 '14 at 08:56
I've read about thet check_output, I tried to use it but it didn't work, will try it again. Thanks. Do you happen to know how to use it with a longer command like the sudo find ... in my first post? – Bosiwow Jul 25 '14 at 09:01
Is this more pythonic, if I don't care about err message?: def lsCommand(): process = subprocess.check_output(['ls','-l']) return process – Bosiwow Jul 25 '14 at 09:08
As I understand, the find example you gave at the end of the post, can't be done with the check_output methode because you're using stdout=subprocess.PIPE and this can cause deadlocks according to the documentation. Am I right? – Bosiwow Jul 25 '14 at 09:12
2

ah, you `wc.communicate()` – Antti Haapala -- Слава Україні Jul 25 '14 at 09:32

score 1 · Answer 2 · answered Jul 25 '14 at 08:50

1

It will not call time.sleep, it will wait till the command complete its execution, it might be 4 second or more.

When you call process.communicate() it will actual make call to command and set the values in process.returncode.

If you will not call process.communicate() then default value of process.returncode is None and what its return when you replace process.communicate() with time.sleep(4).

Popen.returncode

The child return code, set by poll() and wait() (and indirectly by communicate()). A None value indicates that the process hasn’t terminated yet.

A negative value -N indicates that the child was terminated by signal N (Unix only).

answered Jul 25 '14 at 08:50

Nilesh

20,521
16
92
148

If its solve your problem, select it as answer – Nilesh Jul 25 '14 at 08:54
it partially solves the question, but I'm still wondering if this is the cleanest code to write on python 2.7 to get the output from a shell command. Or are there any cleaner ways? – Bosiwow Jul 25 '14 at 09:00

Most pythonic way to get output from a shell command

2 Answers2