How to save an error message from subprocess command to file

Question

When running bash command using subprocess, I might run into situation where the command is not valid. In this case, bash would return an error messsage. How can we catch this message? I would like to save this message to a log file. The following is an example, where I try to list files in a non-existed directory.

try:
    subprocess.check_call(["ls", "/home/non"]) 
    df = subprocess.Popen(["ls", "/home/non"], stdout=subprocess.PIPE)        
    output, err = df.communicate()
    # process outputs
except Exception as error:        
    print error
    sys.exit(1)

Bash would prints "ls: cannot access /home/non: No such file or directory". How can I get this error message? The error caught by the except line is clearly different, it says "Command '['ls', '/home/non']' returned non-zero exit status 2".

I think you need redirect both: i.e. `stdout=subprocess.PIPE, stderr=subprocess.STDOUT` then the error message will be in `err`. — martineau, Apr 11 '15 at 17:13
@martineau Thanks for the suggestion. I just checked, the error message is in output if add "stderr=..." to the Popen. I prefer a way to catch this message by the excpet line, if there are any. Any ideas? — Luke, Apr 11 '15 at 17:46
To do that you need interprocess communication. Since `subprocess.Popen()` can start any kind of other process that would be difficult. About your only general options are `returncode` and pipes. Consider using `subprocess.check_output` instead of `Popen`. — martineau, Apr 11 '15 at 18:05

score 13 · Answer 1 · answered Apr 11 '15 at 18:49

13

"ls: cannot access /home/non: No such file or directory" is generated by ls command, not bash here.

If you want to handle non-existing files using exception handling then use subprocess.check_output():

#!/usr/bin/env python
from subprocess import check_output, STDOUT, CalledProcessError

try:
    output = check_output(['ls', 'nonexistent'], stderr=STDOUT)
except CalledProcessError as exc:
    print(exc.output)
else:
    assert 0

Output

ls: cannot access nonexistent: No such file or directory

answered Apr 11 '15 at 18:49

jfs

399,953
195
994
1,670

Just for clarification purposes, this is a method of accessing some _existing_ subprocess' `stderr` in the global Python session, correct? Instead of sending the error message(s) to a created text file, such as Cunningham's answer describes? – Coolio2654 May 20 '19 at 20:33
@Coolio2654 `stderr` parameter works as usual here. You can pass a file object (with a valid `.fileno()`) if you'd like to save stderr to a file. – jfs May 21 '19 at 13:36

score 8 · Accepted Answer · edited Dec 13 '16 at 02:16

8

You can redirect stderr to a file object:

from subprocess import PIPE, CalledProcessError, check_call, Popen

with open("log.txt", "w") as f:
    try:
        check_call(["ls", "/home/non"], stderr=f)
        df = Popen(["ls", "/home/non"], stdout=PIPE)
        output, err = df.communicate()
    except CalledProcessError as e:
        print(e)
        exit(1)

Output to log.txt:

ls: cannot access /home/non: No such file or directory

If you want the message in the except:

try:
    check_call(["ls", "/home/non"])
    df = Popen(["ls", "/home/non"], stdout=PIPE)
    output, err = df.communicate()
except CalledProcessError as e:
    print(e.message)

For python 2.6 the e.message won't work. You can use a similar version of python 2.7's check_output that will work with python 2.6:

from subprocess import PIPE, CalledProcessError, Popen

def check_output(*args, **kwargs):
    process = Popen(stdout=PIPE, *args, **kwargs)
    out, err = process.communicate()
    ret = process.poll()
    if ret:
        cmd = kwargs.get("args")
        if cmd is None:
            cmd = args[0]
        error = CalledProcessError(ret, cmd)
        error.out = out
        error.message = err
        raise error
    return out

try:
    out = check_output(["ls", "/home"], stderr=PIPE)
    df = Popen(["ls", "/home/non"], stdout=PIPE)
    output, err = df.communicate()
except CalledProcessError as e:
    print(e.message)
else:
    print(out)

edited Dec 13 '16 at 02:16

AnukuL

595
1
7
21

answered Apr 11 '15 at 17:18

Padraic Cunningham

176,452
29
245
321

Is that possible to catch this message by the except line? – Luke Apr 11 '15 at 17:44
@Luke, yes just catch the `CalledProcessError` and print the message – Padraic Cunningham Apr 11 '15 at 17:49
Thanks a lot. Unfortunately, I got this" "DeprecationWarning: BaseException.message has been deprecated as of Python 2.6". – Luke Apr 11 '15 at 17:53
What are you catching? Also what version of python? – Padraic Cunningham Apr 11 '15 at 17:57
I use python 2.6.6, just got an error. test.py:15: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 print e.message – Luke Apr 11 '15 at 18:00
I mean are you using the code exactly as posted. What version of python are you using? – Padraic Cunningham Apr 11 '15 at 18:00
Yes, there is. When I commented out the lines "df = .." and "output ...". I still get the error: "ls: cannot access /home/non: No such file or directory test.py:14: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 print(e.message)" – Luke Apr 11 '15 at 18:07
sorry yes, scratch that I was seeing check_output – Padraic Cunningham Apr 11 '15 at 18:08
Ok, so what happens if you just print e? – Padraic Cunningham Apr 11 '15 at 18:17
It prints "Command '['ls', '/home/non']' returned non-zero exit status 2". – Luke Apr 11 '15 at 18:21
there is no .message for CalledProcessError in python 2.6. I added a version of check_output that should do what you want – Padraic Cunningham Apr 11 '15 at 19:43

ryanjdillon · Answer 3 · 2022-04-29T11:16:40.913

Per the subprocess docs, the command run is now prefered.

Example:

import logging
import subprocess

logger = logging.getLogger(__name__)

try:
    subprocess.run(["ls", "-l"], shell=True, check=True, capture_output=True)
except subprocess.CalledProcessError as err:
    logger.error(f"{err} {err.stderr.decode('utf8')}")

As others have mentioned, if you want to save to a file, you cand use the stdout param to run; however, you may as well use logging to do that, and then just log the error in your method.

How to save an error message from subprocess command to file

3 Answers3

Output

Linked