How to catch exception output from Python subprocess.check_output()?

Question

I'm trying to do a Bitcoin payment from within Python. In bash I would normally do this:

bitcoin sendtoaddress <bitcoin address> <amount>

So for example:

bitcoin sendtoaddress 1HoCUcbK9RbVnuaGQwiyaJGGAG6xrTPC9y 1.4214

If it is successful I get a transaction id as output, but if I try to transfer an amount larger than my bitcoin balance, I get the following output:

error: {"code":-4,"message":"Insufficient funds"}

In my Python program I now try to do the payment as follows:

import subprocess

try:
    output = subprocess.check_output(['bitcoin', 'sendtoaddress', address, str(amount)])
except:
    print "Unexpected error:", sys.exc_info()

If there's enough balance it works fine, but if there's not enough balance sys.exc_info() prints out this:

(<class 'subprocess.CalledProcessError'>, CalledProcessError(), <traceback object at 0x7f339599ac68>)

It doesn't include the error which I get on the command line though. So my question is; how can I get the outputted error ({"code":-4,"message":"Insufficient funds"}) from within Python?

score 177 · Accepted Answer · edited Jul 27 '22 at 06:28

177

According to the subprocess.check_output() docs, the exception raised on error has an output attribute that you can use to access the error details:

try:
    subprocess.check_output(...)
except subprocess.CalledProcessError as e:
    print(e.output)

You should then be able to analyse this string and parse the error details with the json module:

if e.output.startswith('error: {'):
    error = json.loads(e.output[7:]) # Skip "error: "
    print(error['code'])
    print(error['message'])

edited Jul 27 '22 at 06:28

Boris Verkhovskiy

14,854
11
100
103

answered Jul 20 '14 at 11:28

Ferdinand Beyer

64,979
15
154
145

I'm calling a program that outputs something to stdout and then returns 1, but check_output is not capturing it – JorgeeFG Apr 21 '16 at 17:39
@JorgeeFG Then I guess there is something wrong with your program. Please note that the comments section is not the right place to ask new questions. If you need help with your particular problem, click the big "Ask Question" button on the top-right of the page. – Ferdinand Beyer Apr 22 '16 at 07:32
6

In Python 3.5+ you can use `e.stderr` and `e.stdout` instead of `e.output`. – Boris Verkhovskiy Apr 09 '21 at 14:13
The output attribute returns everything from the child process. How can we get only a raised error thrown in the child process to return, without all the output? – David Pesetsky Sep 29 '22 at 13:40

score 60 · Answer 2 · answered Feb 25 '16 at 16:51

60

I don't think the accepted solution handles the case where the error text is reported on stderr. From my testing the exception's output attribute did not contain the results from stderr and the docs warn against using stderr=PIPE in check_output(). Instead, I would suggest one small improvement to J.F Sebastian's solution by adding stderr support. We are, after all, trying to handle errors and stderr is where they are often reported.

from subprocess import Popen, PIPE

p = Popen(['bitcoin', 'sendtoaddress', ..], stdout=PIPE, stderr=PIPE)
output, error = p.communicate()
if p.returncode != 0: 
   print("bitcoin failed %d %s %s" % (p.returncode, output, error))

answered Feb 25 '16 at 16:51

Phil R

761
6
4

I agree that the `stderr` output is very relevant here. An alternative solution would be to use the `run()` function instead (see the [check_output docs](https://docs.python.org/3/library/subprocess.html#subprocess.check_output) how to replace). Because then you can use `e.stderr` from the exception in your error reporting. – Sebastian May 24 '17 at 14:04
2

This should be at the top. – Oleg Oct 08 '17 at 17:23
1

to preserve the apparent order of the output, you could use `stderr=STDOUT` (to merge both streams). – jfs May 23 '18 at 11:38
1

I think it's good to point out that if you do not call `.communicate` then the `.returncode` output will be empty (`None`) – lupodellasleppa May 24 '19 at 07:54
`if p.returncode != 0:` did not work here, because I get `None` on success. Had to use `if p.returncode:` – Giulio Sep 27 '20 at 15:49
2

In Python 3.5+ you can use `.stderr` and `.stdout` instead of `.output` in the accepted answer. – Boris Verkhovskiy Apr 09 '21 at 14:12
This did the job for me. Truly should stay at the top! – Marco Frag Delle Monache Jun 27 '22 at 12:44

score 17 · Answer 3 · answered May 08 '20 at 09:57

As mentioned by @Sebastian the default solution should aim to use run(): https://docs.python.org/3/library/subprocess.html#subprocess.run

Here a convenient implementation (feel free to change the log class with print statements or what ever other logging functionality you are using):

import subprocess

def _run_command(command):
    log.debug("Command: {}".format(command))
    result = subprocess.run(command, shell=True, capture_output=True)
    if result.stderr:
        raise subprocess.CalledProcessError(
                returncode = result.returncode,
                cmd = result.args,
                stderr = result.stderr
                )
    if result.stdout:
        log.debug("Command Result: {}".format(result.stdout.decode('utf-8')))
    return result

And sample usage (code is unrelated, but I think it serves as example of how readable and easy to work with errors it is with this simple implementation):

try:
    # Unlock PIN Card
    _run_command(
        "sudo qmicli --device=/dev/cdc-wdm0 -p --uim-verify-pin=PIN1,{}"
        .format(pin)
    )

except subprocess.CalledProcessError as error:
    if "couldn't verify PIN" in error.stderr.decode("utf-8"):
        log.error(
                "SIM card could not be unlocked. "
                "Either the PIN is wrong or the card is not properly connected. "
                "Resetting module..."
                )
        _reset_4g_hat()
        return

because `result.stderr` contains bytes, should it rather be `if result.stderr.decode('utf-8')`? — Bendemann, Sep 04 '21 at 16:43
Finally, this correctly captures those error messages that get printed on the terminal. — Vitalis, Feb 07 '22 at 06:07

score 12 · Answer 4 · answered Jul 26 '14 at 14:24

12

Trying to "transfer an amount larger than my bitcoin balance" is not an unexpected error. You could use Popen.communicate() directly instead of check_output() to avoid raising an exception unnecessarily:

from subprocess import Popen, PIPE

p = Popen(['bitcoin', 'sendtoaddress', ..], stdout=PIPE)
output = p.communicate()[0]
if p.returncode != 0: 
   print("bitcoin failed %d %s" % (p.returncode, output))

answered Jul 26 '14 at 14:24

jfs

399,953
195
994
1,670

Python encourages an EAFP programming style (Easier to ask for forgiveness than permission), preferring exception handling over 'if' checks in situations like these. – Ferdinand Beyer Jun 26 '15 at 05:45
9

@FerdinandBeyer: EAFP does not apply in this case: you are not making any calls that you wouldn't make otherwise. The code does not have the LBYL structure: `if check(): do()` that you could have replaced with EAFP `try: do() except Error: handle_error()`. The code in the answer inlines `check_output()` function and avoids raising an exception in the `if p.returncode` branch only to catch it at the same level. Avoid cargo-cult programming, think – jfs Jun 26 '15 at 11:48
2

We can also do: `p = Popen(['bitcoin', 'sendtoaddress', ..], stdout=PIPE, stderr=PIPE)` and capture the error message as: `output, error = p.communicate()`. – alper Sep 13 '18 at 11:59
How could I do this also for the commands that are using pipe? @jfs – alper Sep 02 '19 at 11:40
@alper pass the command as a string and add `shell=True` argument: `p = Popen("a | b", shell=True, ..)` – jfs Sep 06 '19 at 18:30
As far as I know, it is now safe to use `shell=True`, I was looking a solution without using `shell=True` @jfs – alper Sep 06 '19 at 20:14
@alper | (pipe) is a shell syntax. You have to use a shell or something that implements the syntax. Unless you have to accept an untrusted input, a shell is ok. https://stackoverflow.com/a/16709666/4279 – jfs Sep 06 '19 at 20:30

score 10 · Answer 5 · answered Dec 02 '21 at 03:58

Since Python 3.5, subprocess.run() supports check argument:

If check is true, and the process exits with a non-zero exit code, a CalledProcessError exception will be raised. Attributes of that exception hold the arguments, the exit code, and stdout and stderr if they were captured.

A simple example that will raise and print out CalledProcessError:

import subprocess
try:
    subprocess.run("exit 1", shell=True, check=True, timeout=15, capture_output=True)
except subprocess.CalledProcessError as e:
    print(e)  # Output: Command 'exit 1' returned non-zero exit status 1.

all of these are also available: `e.cmd` `e.returncode` `e.output` `e.stdout` `e.stderr` — andrew lorien, Feb 24 '22 at 05:43

score 6 · Answer 6 · answered Aug 24 '20 at 23:01

This did the trick for me. It captures all the stdout output from the subprocess(For python 3.8):

from subprocess import check_output, STDOUT
cmd = "Your Command goes here"
try:
    cmd_stdout = check_output(cmd, stderr=STDOUT, shell=True).decode()
except Exception as e:
    print(e.output.decode()) # print out the stdout messages up to the exception
    print(e) # To print out the exception message

score 5 · Answer 7 · answered Mar 10 '17 at 15:42

There are good answers here, but in these answers, there has not been an answer that comes up with the text from the stack-trace output, which is the default behavior of an exception.

If you wish to use that formatted traceback information, you might wish to:

import traceback

try:
    check_call( args )
except CalledProcessError:
    tb = traceback.format_exc()
    tb = tb.replace(passwd, "******")
    print(tb)
    exit(1)

As you might be able to tell, the above is useful in case you have a password in the check_call( args ) that you wish to prevent from displaying.

important to be mindful of logging secrets ^ – mistakeNot Apr 01 '21 at 12:59 — mistakeNot, Apr 01 '21 at 12:59

score 2 · Answer 8 · answered Jun 28 '22 at 20:28

I think most of previous answers are correct, in my case I needed to do this on Windows server and command was a Powershell, for that this worked really nicely for me:

    try:
        
    
        print("inpgoress")           

        cmd_exec="Get-Date"
        print(cmd_aws)

        subprocess.run(['powershell', '-Command', cmd_exec],shell=False,check=True,capture_output=True,text=True,encoding="utf-8")

        

    except Exception as e:
        print(e)
        print("ERROR: something went wrong executing powershell command")
        raise e

score 1 · Answer 9 · answered Dec 26 '19 at 14:40

Based on the answer of @macetw I print the exception directly to stderr in a decorator.

Python 3

from functools import wraps
from sys import stderr
from traceback import format_exc
from typing import Callable, Collection, Any, Mapping


def force_error_output(func: Callable):
    @wraps(func)
    def forced_error_output(*args: Collection[Any], **kwargs: Mapping[str, Any]):
        nonlocal func

        try:
            func(*args, **kwargs)
        except Exception as exception:
            stderr.write(format_exc())
            stderr.write("\n")
            stderr.flush()

            raise exception

    return forced_error_output

Python 2

from functools import wraps
from sys import stderr
from traceback import format_exc


def force_error_output(func):
    @wraps(func)
    def forced_error_output(*args, **kwargs):
        try:
            func(*args, **kwargs)
        except Exception as exception:
            stderr.write(format_exc())
            stderr.write("\n")
            stderr.flush()

            raise exception

    return forced_error_output

Then in your worker just use the decorator

@force_error_output
def da_worker(arg1: int, arg2: str):
    pass

score -1 · Answer 10 · edited Jun 12 '22 at 06:32

The subprocess invoked needs to be told to capture the output in the invoked program and raise the exception. It's simple to do it.

Firstly, Use

subprocess.run() instead of subprocess.call()

Let's assume u wanna python script called "Vijay.py". For raising the exception, use the following;

subprocess.run("py vijay.py", check=True, capture_output=True, shell=True)

The above method then can be put in try and except block to immediately raise the error or can use sys.exit(1) :any non-zero exit is fine

try:
    subprocess.call("py vijay.py", check=True, capture_output=True, shell=True)
except Exception as e:
    print("Exception raised: ", e)

and body of vijay.py can be as follows;

vijay.py

try:
    Your code is here...
except Exception as e:
    sys.exit(1) // or can even use raise Exception("ur own exception to raise:)
enter code here

How to catch exception output from Python subprocess.check_output()?

10 Answers10

Linked