How to check if Popen from subprocess throws an error

Question

I'm writing a pipeline that uses a bunch of bash calls in a particular order.

How can I tell if my command threw an error?

For example, I am running a Java program with subprocess and it doesn't warn me or exit when this process errors.

Is there something in subprocess or in my process object that has this utility?

process = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
process.wait()

When I try the suggestion from Python: "subprocess.Popen" check for success and errors I get the following errors:

In [6]: subprocess.check_call(process)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-6-f8f8752a245f> in <module>
----> 1 subprocess.check_call(process)

~/anaconda/envs/µ_env/lib/python3.6/subprocess.py in check_call(*popenargs, **kwargs)
    284     check_call(["ls", "-l"])
    285     """
--> 286     retcode = call(*popenargs, **kwargs)
    287     if retcode:
    288         cmd = kwargs.get("args")

~/anaconda/envs/µ_env/lib/python3.6/subprocess.py in call(timeout, *popenargs, **kwargs)
    265     retcode = call(["ls", "-l"])
    266     """
--> 267     with Popen(*popenargs, **kwargs) as p:
    268         try:
    269             return p.wait(timeout=timeout)

~/anaconda/envs/µ_env/lib/python3.6/subprocess.py in __init__(self, args, bufsize, executable, stdin, stdout, stderr, preexec_fn, close_fds, shell, cwd, env, universal_newlines, startupinfo, creationflags, restore_signals, start_new_session, pass_fds, encoding, errors)
    707                                 c2pread, c2pwrite,
    708                                 errread, errwrite,
--> 709                                 restore_signals, start_new_session)
    710         except:
    711             # Cleanup if the child failed starting.

~/anaconda/envs/µ_env/lib/python3.6/subprocess.py in _execute_child(self, args, executable, preexec_fn, close_fds, pass_fds, cwd, env, startupinfo, creationflags, shell, p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite, restore_signals, start_new_session)
   1218                 args = [args]
   1219             else:
-> 1220                 args = list(args)
   1221
   1222             if shell:

TypeError: 'Popen' object is not iterable

In [7]: subprocess.check_output(process)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-7-0ec9e7eac1c2> in <module>
----> 1 subprocess.check_output(process)

~/anaconda/envs/µ_env/lib/python3.6/subprocess.py in check_output(timeout, *popenargs, **kwargs)
    334
    335     return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
--> 336                **kwargs).stdout
    337
    338

~/anaconda/envs/µ_env/lib/python3.6/subprocess.py in run(input, timeout, check, *popenargs, **kwargs)
    401         kwargs['stdin'] = PIPE
    402
--> 403     with Popen(*popenargs, **kwargs) as process:
    404         try:
    405             stdout, stderr = process.communicate(input, timeout=timeout)

~/anaconda/envs/µ_env/lib/python3.6/subprocess.py in __init__(self, args, bufsize, executable, stdin, stdout, stderr, preexec_fn, close_fds, shell, cwd, env, universal_newlines, startupinfo, creationflags, restore_signals, start_new_session, pass_fds, encoding, errors)
    707                                 c2pread, c2pwrite,
    708                                 errread, errwrite,
--> 709                                 restore_signals, start_new_session)
    710         except:
    711             # Cleanup if the child failed starting.

~/anaconda/envs/µ_env/lib/python3.6/subprocess.py in _execute_child(self, args, executable, preexec_fn, close_fds, pass_fds, cwd, env, startupinfo, creationflags, shell, p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite, restore_signals, start_new_session)
   1218                 args = [args]
   1219             else:
-> 1220                 args = list(args)
   1221
   1222             if shell:

TypeError: 'Popen' object is not iterable

score 8 · Accepted Answer · edited Sep 03 '19 at 06:57

Both check_call and check_output should be passed a list of commands to be run by the command system (the same list of commands that would be sent to Popen). Both of these are blocking calls, meaning that Python will wait until they finish to run more code.

Using Popen sends the list of commands to the command system, but does not block further execution of Python code. You can check in on the process using the .poll method of the Popen object, or you can make a blocking call using .communicate which will return a tuple of the standard out and standard error streams.

Assuming you want the results of the executed command, and the command will properly report an error to the error stream, you can use:

process = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, err = process.communicate()

if err:
     print('The process raised an error:', err.decode())

Here are some examples:

Using `Popen`:

import subprocess

# first a command that works correctly
proc = subprocess.Popen(['ls', '-a'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, err = proc.communicate()
if not err:
    print('--No errors--\n', out.decode())
else:
    print('--Error--\n', err.decode())

# prints:
--No errors--
anaconda3
Desktop
Documents
Downloads

# next a command, which generates an error. The `-w` switch is invalid
proc = subprocess.Popen(['ls', '-w'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, err = proc.communicate()
if not err:
    print('--No errors--\n', out.decode())
else:
    print('--Error--\n', err.decode())

# prints:
--Error--
 ls: option requires an argument -- 'w'
Try 'ls --help' for more information.

Using `check_call`

check_call will raise a Python exception if the return code from the command system is not 0.

# first with a working command:
ret_code = subprocess.check_call(['ls', '-a'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
ret_code
# returns:
0

# and now with the command that generates an error:
ret_code = subprocess.check_call(['ls', '-w'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)

# raises an exception:
---------------------------------------------------------------------------
CalledProcessError                        Traceback (most recent call last)
<ipython-input-25-3cd5107991a2> in <module>()
----> 1 ret_code = subprocess.check_call(['ls', '-w'], stdout=subprocess.PIPE, 
        stderr=subprocess.PIPE)

~/anaconda3/lib/python3.6/subprocess.py in check_call(*popenargs, **kwargs)
    289         if cmd is None:
    290             cmd = popenargs[0]
--> 291         raise CalledProcessError(retcode, cmd)
    292     return 0
    293

CalledProcessError: Command '['ls', '-w']' returned non-zero exit status 2.

To handle the exception, use a try/except block.

try:
    ret_code = subprocess.check_call(['ls', '-w'], stdout=subprocess.PIPE, 
        stderr=subprocess.PIPE)
except subprocess.CalledProcessError as e:
    ret_code = e.returncode
    print('An error occurred.  Error code:', ret_code)

# prints:
An error occurred.  Error code: 2

Using `check_output`

check_output is very similar to check_call in that it will raise a Python exception if the return code from the command system is not 0. However, if the return code is 0, it will return the output in the standard output stream.

# just with the error process this time
try:
    ret_code = subprocess.check_output(['ls', '-w'], stdout=subprocess.PIPE, 
        stderr=subprocess.PIPE)
except subprocess.CalledProcessError as e:
    ret_code = e.returncode
    print('An error occurred.  Error code:', ret_code)

# prints:
An error occurred.  Error code: 2

So this can still be used after I run `process.wait()` correct? — O.rka, Jan 16 '19 at 00:28
Yes, `process.wait()` adds a blocking call until the process finishes, but does not return the stdout and stderr streams. — James, Jan 16 '19 at 00:33
Thanks a lot! This is a great answer and really describes what is going on. — O.rka, Jan 16 '19 at 00:54

How to check if Popen from subprocess throws an error

1 Answers1

Here are some examples:

Using Popen:

Using check_call

Using check_output

Using `Popen`:

Using `check_call`

Using `check_output`